Manifold Alignment for Semantically Aligned Style Transfer

Related tags

Deep LearningMAST
Overview

Manifold Alignment for Semantically Aligned Style Transfer

[Paper]

res1 GUI Demo

Getting Started

MAST has been tested on CentOS 7.6 with python >= 3.6. It supports both GPU and CPU inference. If you don't have a suitable device, try running our Colab demo.

Clone the repo:

git clone https://github.com/NJUHuoJing/MAST.git

prepare the checkpoints:

cd MAST
chmod 777 scripts/prepare_data.sh
scripts/prepare_data.sh

Install the requirements:

conda create -n mast-env python=3.6
conda activate mast-env
pip install -r requirements.txt

# If you want to use post smoothing as the same as PhotoWCT, then install the requirements below;
# You can also just skip it to use fast post smoothing, remember to change cfg.TEST.PHOTOREALISTIC.FAST_SMOOTHING=true
pip install -U setuptools
pip install cupy
pip install pynvrtc

Running the Demo

Artistic style transfer

First set MAST_CORE.ORTHOGONAL_CONSTRAINT=false in configs/config.yaml. Then use the script test_artistic.py to generate the artistic stylized image by following the command below:

# not use seg
python test_artistic.py --cfg_path configs/config.yaml --content_path data/default/content/4.png --style_path data/default/style/4.png --output_dir results/test/default

# use --content_seg_path and --style_seg_path to user edited style transfer
python test_artistic.py --cfg_path configs/config.yaml --content_path data/default/content/4.png --style_path data/default/style/4.png --output_dir results/test/default --content_seg_path data/default/content_segmentation/4.png --style_seg_path data/default/style_segmentation/4.png --seg_type labelme --resize 512

Photo-realistic style transfer

First set MAST_CORE.ORTHOGONAL_CONSTRAINT=true in configs/config.yaml. Then use the script test_photorealistic.py to generate the photo-realistic stylized image by following the command below:

# not use seg
python test_photorealistic.py --cfg_path configs/config.yaml --content_path data/photo_data/content/in1.png --style_path data/photo_data/style/tar1.png --output_dir results/test/photo --resize 512

# or use --content_seg_path and --style_seg_path to user edited style transfer
python test_photorealistic.py --cfg_path configs/config.yaml --content_path data/photo_data/content/in1.png --style_path data/photo_data/style/tar1.png --output_dir results/test/photo --content_seg_path data/photo_data/content_segmentation/in1.png --style_seg_path data/photo_data/style_segmentation/tar1.png --seg_type dpst --resize 512

GUI For Artistic style transfer and User Editing

We provide a gui for user-controllable artistic image stylization. Just use the command below to run test_gui.py

python test_gui.py --cfg_path configs/config.yaml

Features

  1. You can use different colors to control the style transfer in different semantic areas.
  2. The button Expand and Expand num respectively control whether to expand the selected semantic area and the degree of expansion.

See the gif demo for more details.

Google Colab

If you do not have a suitable environment to run this project then you could give Google Colab a try. It allows you to run the project in the cloud, free of charge. You may try our Colab demo using the notebook we have prepared: Colab Demo

Citation

@inproceedings{huo2021manifold,
    author = {Jing Huo and Shiyin Jin and Wenbin Li and Jing Wu and Yu-Kun Lai and Yinghuan Shi and Yang Gao},
    title = {Manifold Alignment for Semantically Aligned Style Transfer},
    booktitle = {IEEE International Conference on Computer Vision},
    pages     = {14861-14869},
    year = {2021}
}

References

  • The post smoothing module is borrowed from PhotoWCT
A self-supervised learning framework for audio-visual speech

AV-HuBERT (Audio-Visual Hidden Unit BERT) Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction Robust Self-Supervised A

Meta Research 431 Jan 07, 2023
KwaiRec: A Fully-observed Dataset for Recommender Systems (Density: Almost 100%)

KuaiRec: A Fully-observed Dataset for Recommender Systems (Density: Almost 100%) KuaiRec is a real-world dataset collected from the recommendation log

Chongming GAO (高崇铭) 70 Dec 28, 2022
Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

NeX: Real-time View Synthesis with Neural Basis Expansion Project Page | Video | Paper | COLAB | Shiny Dataset We present NeX, a new approach to novel

538 Jan 09, 2023
(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

NeRF--: Neural Radiance Fields Without Known Camera Parameters Project Page | Arxiv | Colab Notebook | Data Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min

Active Vision Laboratory 411 Dec 26, 2022
Code for weakly supervised segmentation of a single class

SingleClassRL Implementation of weak single object segmentation from paper "Regularized Loss for Weakly Supervised Single Class Semantic Segmentation"

16 Nov 14, 2022
Trading Strategies for Freqtrade

Freqtrade Strategies Strategies for Freqtrade, developed primarily in a partnership between @werkkrew and @JimmyNixx from the Freqtrade Discord. Use t

Bryan Chain 242 Jan 07, 2023
[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.

MVSNeRF Project page | Paper This repository contains a pytorch lightning implementation for the ICCV 2021 paper: MVSNeRF: Fast Generalizable Radiance

Anpei Chen 529 Dec 30, 2022
Overview of architecture and implementation of TEDS-Net, as described in MICCAI 2021: "TEDS-Net: Enforcing Diffeomorphisms in Spatial Transformers to Guarantee TopologyPreservation in Segmentations"

TEDS-Net Overview of architecture and implementation of TEDS-Net, as described in MICCAI 2021: "TEDS-Net: Enforcing Diffeomorphisms in Spatial Transfo

Madeleine K Wyburd 14 Jan 04, 2023
SynNet - synthetic tree generation using neural networks

SynNet This repo contains the code and analysis scripts for our amortized approach to synthetic tree generation using neural networks. Our model can s

Wenhao Gao 60 Dec 29, 2022
Fast EMD for Python: a wrapper for Pele and Werman's C++ implementation of the Earth Mover's Distance metric

PyEMD: Fast EMD for Python PyEMD is a Python wrapper for Ofir Pele and Michael Werman's implementation of the Earth Mover's Distance that allows it to

William Mayner 433 Dec 31, 2022
E-RAFT: Dense Optical Flow from Event Cameras

E-RAFT: Dense Optical Flow from Event Cameras This is the code for the paper E-RAFT: Dense Optical Flow from Event Cameras by Mathias Gehrig, Mario Mi

Robotics and Perception Group 71 Dec 12, 2022
A library for low-memory inferencing in PyTorch.

Pylomin Pylomin (PYtorch LOw-Memory INference) is a library for low-memory inferencing in PyTorch. Installation ... Usage For example, the following c

3 Oct 26, 2022
Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters.

Joint Unsupervised Learning (JULE) of Deep Representations and Image Clusters. Overview This project is a Torch implementation for our CVPR 2016 paper

Jianwei Yang 278 Dec 25, 2022
Pytorch implementation of face attention network

Face Attention Network Pytorch implementation of face attention network as described in Face Attention Network: An Effective Face Detector for the Occ

Hooks 312 Dec 09, 2022
Dirty Pixels: Towards End-to-End Image Processing and Perception

Dirty Pixels: Towards End-to-End Image Processing and Perception This repository contains the code for the paper Dirty Pixels: Towards End-to-End Imag

50 Nov 18, 2022
Repo for 2021 SDD assessment task 2, by Felix, Anna, and James.

SoftwareTask2 Repo for 2021 SDD assessment task 2, by Felix, Anna, and James. File/folder structure: helloworld.py - demonstrates various map backgrou

3 Dec 13, 2022
Create UIs for prototyping your machine learning model in 3 minutes

Note: We just launched Hosted, where anyone can upload their interface for permanent hosting. Check it out! Welcome to Gradio Quickly create customiza

Gradio 11.7k Jan 07, 2023
Python utility to generate filesystem content for Obsidian.

Security Vault Generator Quickly parse, format, and output common frameworks/content for Obsidian.md. There is a strong focus on MITRE ATT&CK because

Justin Angel 73 Dec 02, 2022
A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis

A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis This is the pytorch implementation for our MICCAI 2021 paper. A Mul

Jiarong Ye 7 Apr 04, 2022
Creating multimodal multitask models

Fusion Brain Challenge The English version of the document can be found here. Обновления 01.11 Мы выкладываем пример данных, аналогичных private test

Sber AI 43 Nov 28, 2022