The source code of CVPR 2019 paper "Deep Exemplar-based Video Colorization".

Last update: Dec 27, 2022

Overview

Deep Exemplar-based Video Colorization (Pytorch Implementation)

Paper | Pretrained Model | Youtube video 🔥 | Colab demo

Deep Exemplar-based Video Colorization, CVPR2019

Bo Zhang^1,3, Mingming He^1,5, Jing Liao², Pedro V. Sander¹, Lu Yuan⁴, Amine Bermak¹, Dong Chen³
¹Hong Kong University of Science and Technology,²City University of Hong Kong, ³Microsoft Research Asia, ⁴Microsoft Cloud&AI, ⁵USC Institute for Creative Technologies

Prerequisites

Python 3.6+
Nvidia GPU + CUDA, CuDNN

Installation

First use the following commands to prepare the environment:

conda create -n ColorVid python=3.6
source activate ColorVid
pip install -r requirements.txt

Then, download the pretrained models from this link, unzip the file and place the files into the corresponding folders:

video_moredata_l1 under the checkpoints folder
vgg19_conv.pth and vgg19_gray.pth under the data folder

Data Preparation

In order to colorize your own video, it requires to extract the video frames, and provide a reference image as an example.

Place your video frames into one folder, e.g., ./sample_videos/v32_180
Place your reference images into another folder, e.g., ./sample_videos/v32

If you want to automatically retrieve color images, you can try the retrieval algorithm from this link which will retrieve similar images from the ImageNet dataset. Or you can try this link on your own image database.

Test

python test.py --image-size [image-size] \
               --clip_path [path-to-video-frames] \
               --ref_path [path-to-reference] \
               --output_path [path-to-output]

We provide several sample video clips with corresponding references. For example, one can colorize one sample legacy video using:

python test.py --clip_path ./sample_videos/clips/v32 \
               --ref_path ./sample_videos/ref/v32 \
               --output_path ./sample_videos/output

Note that we use 216*384 images for training, which has aspect ratio of 1:2. During inference, we scale the input to this size and then rescale the output back to the original size.

Train

We also provide training code for reference. The training can be started by running:

python --data_root [root of video samples] \
       --data_root_imagenet [root of image samples] \
       --gpu_ids [gpu ids] \

We do not provide the full video dataset due to the copyright issue. For image samples, we retrieve semantically similar images from ImageNet using this repository. Still, one can refer to our code to understand the detailed procedure of augmenting the image dataset to mimic the video frames.

Comparison with State-of-the-Arts

More results

⭐ Please check our Youtube demo for results of video colorization.

Citation

If you use this code for your research, please cite our paper.

@inproceedings{zhang2019deep,
title={Deep exemplar-based video colorization},
author={Zhang, Bo and He, Mingming and Liao, Jing and Sander, Pedro V and Yuan, Lu and Bermak, Amine and Chen, Dong},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={8052--8061},
year={2019}
}

Old Photo Restoration 🔥

If you are also interested in restoring the artifacts in the legacy photo, please check our recent work, bringing old photo back to life.

@inproceedings{wan2020bringing,
title={Bringing Old Photos Back to Life},
author={Wan, Ziyu and Zhang, Bo and Chen, Dongdong and Zhang, Pan and Chen, Dong and Liao, Jing and Wen, Fang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={2747--2757},
year={2020}
}

License

This project is licensed under the MIT license.

Comments

I met "CUDA error: an illegal memory access was encountered" problem

When i tested your pre-trained model, i met the problem called "CUDA error: an illegal memory access was encountered", can you provide the version of your CUDA, cudnn and pytorch

opened by Zonobia-A 3
Could i apply this method to image colorization

Hi, could i apply this method to image colorization and remove the temporal consistency loss? BTW, how to get the pairs.txt/pairs_mid.txt/pairs_bad.txt used in videoloader_imagenet.py?

opened by buptlj 2
video size very low

video colorization very good and very impressive . but render image low size 768x432 . and video size also same. how to in increase the image size and video size. thank you..

opened by srinivas68 2
training command is wrong.

The original training command is python --data_root [root of video samples] \ --data_root_imagenet [root of image samples] \ --gpu_ids [gpu ids] \ Maybe it should be python train.py --data_root [root of video samples] \ --data_root_imagenet [root of image samples] \ --gpu_ids [gpu ids] \ ?

opened by Horizon2333 1
There seems a bug ofr feature centering with x_features - y_features.mean

There seems a bug ofr feature centering with x_features - y_features.mean which I think should be x_features - x_features.mean https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization/blob/37639748f12dfecbb0a3fe265b533887b5fe46ce/models/ContextualLoss.py#L100

opened by JerryLeolfl 1
the test code

Thanks for your great work! I have a question when i run test.py. Why don't you extract the feature of inference image out of the for loop. I haven't found any difference.

opened by buptlj 1
CUDA OOM

Hello, I am running a 4gb nvidia GPU. Is that enough for inference? I try to run on ubuntu 18.04 as well as windows but always get a Out of memory error eventually. Sometimes happen after 2nd image and sometimes after 5th. This is 1080p video.

opened by quocthaitang 1
illustrative training data

Could you please release a tiny illustrative training dataset, such that the preparation of a custom training data can be easily followed. Currently, it is not easy to prepare a custom training data by reading the train.py. or could you please give a further explanation of the following fields? ( image1_name, image2_name, reference_video_name, reference_video_name1, reference_name1, reference_name2, reference_name3, reference_name4, reference_name5, reference_gt1, reference_gt2, reference_gt3, ) Thank you very much.

opened by davyfeng 1
Runtime error

Getting a runtime error when running the test cells at the 'first we visualize the input video'. I'm not good with code, but this is the first time I've experienced this issue with this wonderful program. No cells execute after this error. I've attached a screenshot.

opened by StevieMaccc 0
Test result problem

At present, after training, it is found that the generated test image is effective, but the color saturation is very low. Is it because of the colored model or other reasons? I'm looking forward to your reply！！！

opened by songyn95 0
Training has little effect

Hello, I read in the paper that "we train the network for 10 epichs with a batch size of 40 pairs of video frames. " Is it effective after only 10 iterations? Is your data 768 videos, 25 frames per video? I only train one video at present, epoch=40, but I find that it has little effect. What may be the reason?

opened by songyn95 0
Error 404 - Important files missing

I was working with the Colab program and there appears to be important models / files missing. As a result the program has ceased to function. I've brough to the designers attention so hopefully will be resolved.

opened by StevieMaccc 1
CUDA device error "module 'torch._C' has no attribute '_cuda_setDevice'" when running test.py

Hi !

Trying out test.py results in the following error:

Traceback (most recent call last): File "test.py", line 26, in <module> torch.cuda.set_device(0) File "C:\Users\natha\anaconda3\envs\ColorVid\lib\site-packages\torch\cuda\__init__.py", line 311, in set_device torch._C._cuda_setDevice(device) AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'

I tried installing pytorch manually using their tool https://pytorch.org/get-started/locally/ (with CUDA 11.6) but that doesn't resolve the issue.

Can someone help me understand what is going on ? Thanks !!

opened by FoxTrotte 4
Questions about the test phase
Thanks for your outstanding work! I have some questions when I read it.

What are the settings when you test this video model on image colorization which used for comparing with other image colorization methods?

Could you please give me a url about your video testset (116 video clips collected from Videvo)? Thanks again for your attention.
opened by JerryLeolfl 0
It seems not correct of the code in TestTransforms.py line 341

https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization/blob/37639748f12dfecbb0a3fe265b533887b5fe46ce/lib/TestTransforms.py#L341 it seems a repeated define of call

opened by JerryLeolfl 0
Wrong output resolution

Processing 4x3 video 912x720 outputs cropped and downscaled 16x9 768x432. Playing around "python test.py --image-size [image-size] " doesn't help My be I don't properly specify an arguments? So, what the the proper use of --image-size [image-size] in order to get 912x720? Greatly appreciate for suggesting.

opened by semel1 5

Releases(v1.0)

v1.0(Dec 1, 2020)

Source code(tar.gz)
Source code(zip)
colorization_checkpoint.zip(770.16 MB)

Owner

Bo Zhang

Researcher

GitHub Repository https://arxiv.org/abs/1906.09909

Predicting path with preference based on user demonstration using Maximum Entropy Deep Inverse Reinforcement Learning in a continuous environment

Preference-Planning-Deep-IRL Introduction Check my portfolio post Dependencies Gym stable-baselines3 PyTorch Usage Take Demonstration python3 record.

9 Oct 26, 2022

Realistic lighting in ursina!

Ursina Lighting Realistic lighting in ursina! If you want to have realistic lighting in ursina, import the UrsinaLighting.py in your project and use t

17 Jul 07, 2022

ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation

ClevrTex This repository contains dataset generation code for ClevrTex benchmark from paper: ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi

26 Dec 21, 2022

Deep Learning Package based on TensorFlow

White-Box-Layer is a Python module for deep learning built on top of TensorFlow and is distributed under the MIT license. The project was started in M

7 Dec 27, 2021

PyMove is a Python library to simplify queries and visualization of trajectories and other spatial-temporal data

Use PyMove and go much further Information Package Status License Python Version Platforms Build Status PyPi version PyPi Downloads Conda version Cond

64 Nov 15, 2022

High performance distributed framework for training deep learning recommendation models based on PyTorch.

PERSIA (Parallel rEcommendation tRaining System with hybrId Acceleration) is developed by AI 340 Dec 30, 2022

An Industrial Grade Federated Learning Framework

DOC | Quick Start | 中文 FATE (Federated AI Technology Enabler) is an open-source project initiated by Webank's AI Department to provide a secure comput

4.8k Jan 09, 2023

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-ba

370 Dec 27, 2022

Multi-Agent Reinforcement Learning (MARL) method to learn scalable control polices for multi-agent target tracking.

scalableMARL Scalable Reinforcement Learning Policies for Multi-Agent Control CD. Hsu, H. Jeong, GJ. Pappas, P. Chaudhari. "Scalable Reinforcement Lea

17 Nov 17, 2022

Python code to generate art with Generative Adversarial Network

GAN_Canvas_Maker Generating Art using Generative Adversarial Network (GAN) Python code to generate art with Generative Adversarial Network: https://to

10 Aug 22, 2022

Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

302 Dec 14, 2022

Fake News Detection Using Machine Learning Methods

Fake-News-Detection-Using-Machine-Learning-Methods Fake news is always a real and dangerous issue. However, with the presence and abundance of various

1 Jan 11, 2022

MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens

MSG-Transformer Official implementation of the paper MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens, by Jiemin

68 Nov 16, 2022

QRec: A Python Framework for quick implementation of recommender systems (TensorFlow Based)

Introduction QRec is a Python framework for recommender systems (Supported by Python 3.7.4 and Tensorflow 1.14+) in which a number of influential and

1.4k Jan 01, 2023

Scripts of Machine Learning Algorithms from Scratch. Implementations of machine learning models and algorithms using nothing but NumPy with a focus on accessibility. Aims to cover everything from basic to advance.

Algo-ScriptML Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The goal of this project is not t

81 Nov 26, 2022