SynergyNet

3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

Cho-Ying Wu, Qiangeng Xu, Ulrich Neumann, CGIT Lab at University of Souther California

[paper] [project page]

This paper supersedes the previous version of M3-LRN.

Advantages:

♦ SOTA on all 3D facial alignment, face orientation estimation, and 3D face modeling.

♦ Fast inference with 3000fps on a laptop RTX 2080 Ti.

♦ Simple implementation with only widely used operations.

Evaluation (This project is built/tested on Python 3.8 and PyTorch 1.9)

1. Clone

   git clone https://github.com/choyingw/SynergyNet

   cd SynergyNet

2. Use conda

   conda create --name SynergyNet

   conda activate SynergyNet

3. Install pre-requisite common packages

   PyTorch 1.9 (should also be compatiable with 1.0+ versions), Opencv, Scipy, Matplotlib

4. Prepare data

Download data [here] and [here]. Extract these data under the repo root.

These data are processed from [3DDFA] and [FSA-Net].

Download pretrained weights [here]. Put the model under 'models/'

5. Benchmarking

   python benchmark.py -w pretrained/best.pth.tar

Print-out results and visualization under 'results/' (see 'demo/' for some sample references) are shown.

TODO

• Single-Image inference
• Add a renderer and 3D face output
• Training script
• Texture synthesis in the supplementary

More Results

Facial alignemnt on AFLW2000-3D (NME of facial landmarks):

Face orientation estimation on AFLW2000-3D (MAE of Euler angles):

Results on artistic faces:

Related Project

[Voice2Mesh] (analysis on relation for voice and 3D face)

Acknowledgement

The project is developed on [3DDFA] and [FSA-Net]. Thank them for their wonderful work.