Long Expressive Memory (LEM)

Last update: Dec 17, 2022

Overview

Long Expressive Memory for Sequence Modeling

This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling

Requirements

To setup the environment, install the requirements using python 3.7.10 with

pip install -r requirements.txt

Experiments

This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:

Adding Task
EigenWorms
FitzHugh-Nagumo
Google12 (V2)
Heart rate prediction
noisy CIFAR-10
Permuted Sequential MNIST
PennTree Bank (word- and char-level)
Sequential MNIST

All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.

Results

The results of LEM for each of the experiments are:

Experiment	Result
EigenWorms	92.3% mean test accuracy
FitzHugh-Nagumo	0.002 test L^2 loss
Google12 (V2)	95.7% test accuracy
Heart rate prediction	0.85 test L^2 loss
noisy CIFAR-10	60.5% test accuracy
Permuted Sequential MNIST	96.6% test accuracy
PennTree Bank char-level (single layer)	1.25 test bits-per-character
PennTree Bank word-level (single layer)	72.8 test perplexity
Sequential MNIST	99.5% test accuracy

Long Expressive Memory (LEM)

Related tags

Overview

Long Expressive Memory for Sequence Modeling

Requirements

Experiments

Results

Owner

Konstantin Rusch

CR-Fill: Generative Image Inpainting with Auxiliary Contextual Reconstruction. ICCV 2021

Code for "Universal inference meets random projections: a scalable test for log-concavity"

Implementation of Retrieval-Augmented Denoising Diffusion Probabilistic Models in Pytorch

Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

Image segmentation with private İstanbul Dataset

Code for Parameter Prediction for Unseen Deep Architectures (NeurIPS 2021)

Repo for parser tensorflow(.pb) and tflite(.tflite)

This repository contains demos I made with the Transformers library by HuggingFace.

This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

A PyTorch-based library for semi-supervised learning

Neural Re-rendering for Full-frame Video Stabilization

Fast Scattering Transform with CuPy/PyTorch

The final project of "Applying AI to EHR Data" of "AI for Healthcare" nanodegree - Udacity.

Predict and time series avocado hass

An efficient and effective learning to rank algorithm by mining information across ranking candidates. This repository contains the tensorflow implementation of SERank model. The code is developed based on TF-Ranking.

Demonstrational Session git repo for H SAF User Workshop (28/1)

Interactive Terraform visualization. State and configuration explorer.

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in clustering (CVPR2021)

SPT_LSA_ViT - Implementation for Visual Transformer for Small-size Datasets

Source code of our BMVC 2021 paper: AniFormer: Data-driven 3D Animation with Transformer