Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation

The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

Anchor-free Oriented Proposal Generator for Object Detection

Converting CPT to bert form for use

The code repository for "PyCIL: A Python Toolbox for Class-Incremental Learning" in PyTorch.

Article Reranking by Memory-enhanced Key Sentence Matching for Detecting Previously Fact-checked Claims.

This is an official pytorch implementation of Fast Fourier Convolution.

Readings for "A Unified View of Relational Deep Learning for Polypharmacy Side Effect, Combination Therapy, and Drug-Drug Interaction Prediction."

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Creating Artificial Life with Reinforcement Learning

This is a deep learning-based method to segment deep brain structures and a brain mask from T1 weighted MRI.

Source code of "Hold me tight! Influence of discriminative features on deep network boundaries"

Implementation of "StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis"

Model parallel transformers in Jax and Haiku

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling @ INTERSPEECH 2021 Accepted

Experimenting with computer vision techniques to generate annotated image datasets from gameplay recordings automatically.

Self Governing Neural Networks (SGNN): the Projection Layer

😮The official implementation of "CoNeRF: Controllable Neural Radiance Fields" 😮

CAR-API: Cityscapes Attributes Recognition API

Aligning Latent and Image Spaces to Connect the Unconnectable