A free, multiplatform SDK for real-time facial motion capture using blendshapes, and rigid head pose in 3D space from any RGB camera, photo, or video.

Last update: Dec 27, 2022

Overview

mocap4face by Facemoji

mocap4face by Facemoji is a free, multiplatform SDK for real-time facial motion capture based on Facial Action Coding System or (FACS). It provides real-time FACS-derived blendshape coefficients, and rigid head pose in 3D space from any mobile camera, webcam, photo, or video enabling live animation of 3D avatars, digital characters, and more.

After fetching the input from one of the mentioned sources, mocap4face SDK produces data in ARKit-compatible blendshapes, i.e., morph targets weight values as a per-frame expression shown in the video below. Useful for, e.g., animating a 2D or 3D avatar in a way that mimics the user's facial expressions in real-time à la Apple Memoji but without the need of a hardware-based TrueDepth Camera.

With mocap4face, you can drive live avatars, build Snapchat-like lenses, AR experiences, face filters that trigger actions, VTubing apps, and more with as little energy impact and CPU/GPU use as possible. As an example, check out how the popular avatar live-streaming app REALITY is using our SDK.

Please ⭐ Star us on GitHub—it motivates us a lot!

📋 Table of Content

Tech Specs
Installation
Use Cases
License
Links

🤓 Tech Specs

iOS, Android, Web (coming soon)
3D reprojection to the input photo/video
Platform-suited API and packaging with internal optimizations
Simultaneous back and front camera support

✨ Key Features

42 tracked facial expressions via blendshapes
Eye tracking including eye gaze vector
Tongue tracking
Light & fast, just 3MB ML model size
≤ ±50° pitch, ≤ ±40° yaw and ≤ ±30° roll tracking coverage

🤳 Input

All RGB camera
Photo
Video

📦 Output

ARKit-compatible blendshapes
Head position and scale in 2D and 3D
Head rotation in world coordinates

⚡ Performance

50 FPS on Pixel 4
60 FPS on iPhone SE (1st gen)
90 FPS on iPhone X or newer

💿 Installation

Prerequisites

Get a new developer account at studio.facemoji.co
Generate a unique API key for your app
Paste the API key to your source code

iOS

Open the sample XCode project and run the demo on your iOS device
To use the SDK in your project, either use the bundled XCFramework directly or use the Swift Package manager (this repository also serves as a Swift PM repository)

Android

Open the sample project in Android Studio and run the demo on your Android device
Add this repository to the list of your Maven repositories in your root build.gradle, for example:

allprojects {
    repositories {
        google()
        mavenCentral()
        // Any other repositories here...

        maven {
            name = "Facemoji"
            url = uri("https://facemoji.jfrog.io/artifactory/default-maven-local/")
        }
    }
}

To use the SDK in your project, add implementation 'co.facemoji:mocap4face:0.1.0' to your Gradle dependencies

🚀 Use Cases

Live avatar experiences
Snapchat-like lense
AR experiences
VTubing apps
Live streaming apps
Face filters
AR games with facial triggers
Beauty AR
Virtual try-on
Play to earn games

❤️ Links

📄 License

This library is provided under the Facemoji SDK License Agreement—see LICENSE.

The sample code in this repository is provided under the Facemoji Samples License

Notices

OSS used in mocap4face SDK:

Comments

Mainactivity re-opened cannot be traced
Only the first open MainActivity can be tracked. when back to SplashActivity and reopen Mainactivity the track result is null.

The test code is as follows（modify SplashAtivity）：

override fun onResume() { super.onResume() Timer("Splash", false).schedule(5000) { startMainActivity() } }
bug
opened by cike978 9
can not find any documents about how to build a app using facemoji SDK

i want to build a app like the offical facemoji app, but the demo code is so simple, the SDK even has no document.

can anyone tell me how to use the SDK to build a facemoji app ?

opened by danlanxiaohei 3
Fix "artifact not found for target 'mocap4face'" error
Issue

I got "artifact not found for target 'mocap4face'" error when I tried to load mocap4face Swift Package by Xcode 13.3

The target name should be exactly the same as the framework name (case sensitive)

Changes

Change the target name from mocap4face to Mocap4Face to match the framework name
opened by magicien 2
Any plans for Windows & MacOS Support?

I'm guessing if this is already cross-platform on Android, Web, and iOS that this an upcoming features. Just curious since we'd love to use this everywhere if possible

opened by aaronn 2
[Question]: Is this Lib compatible with react-native?

It would be incredibly awesome to include this library in my React-Native start-up App but I wonder whether there's React-Native compatible SDK?

Thanks!!!
enhancement

opened by shamxeed 2
OpenGLResizer20: setup OES texture: glError 1282: invalid operation

使用 Android 版本打开 Activity 正常，关闭页面再次启用的时候出现 OpenGLResizer20: setup OES texture: glError 1282: invalid operation

onCameraImageArkitLink(cameraTexture: OpenGLTexture)

cameraTexture = null

opened by pengzhenkun 1
Stop face tracking

Hello. Thanks for the great face tracking library!

How can I stop tracking? Something like a dispose/destroy function?

I am using this library in web (js)

opened by Nawarius 1
Status of macOS build?

Has any new work been done to bring this to the Mac via catalyst?

We primarily just want to be able to use the full camera width when running under M1 catalyst. We don't need the full feature-set, but macOS forces "non-mac" frameworks to use a cropped iPhone-like camera view even on the Mac which has a much wider view.

opened by aaronn 0
Problems with one eye open and the other closed

Platform Android With one eye open and the other closed, the output is eyeBlink_L 0.7 and eyeBlink_R 0.5 Expected result is eyeBlink_L 0.7 and eyeBlink_R 0
bug

opened by will4621d 2

Releases(0.3.0)

Owner

Facemoji

Avatars for apps and games.

GitHub Repository

A package, and script, to perform imaging transcriptomics on a neuroimaging scan.

Imaging Transcriptomics Imaging transcriptomics is a methodology that allows to identify patterns of correlation between gene expression and some prop

10 Dec 27, 2022

Denoising Diffusion Implicit Models

Denoising Diffusion Implicit Models (DDIM) Jiaming Song, Chenlin Meng and Stefano Ermon, Stanford Implements sampling from an implicit model that is t

465 Jan 05, 2023

AITom is an open-source platform for AI driven cellular electron cryo-tomography analysis.

AITom Introduction AITom is an open-source platform for AI driven cellular electron cryo-tomography analysis. AITom is originated from the tomominer l

93 Jan 02, 2023

ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab

AliceMind AliceMind: ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab This repository provides pre-trained encode

1.4k Jan 01, 2023

Simple-Image-Classification - Simple Image Classification Code (PyTorch)

Simple-Image-Classification Simple Image Classification Code (PyTorch) Yechan Kim This repository contains: Python3 / Pytorch code for multi-class ima

8 Oct 29, 2022

Continual Learning of Long Topic Sequences in Neural Information Retrieval

ContinualPassageRanking Repository for the paper "Continual Learning of Long Topic Sequences in Neural Information Retrieval". In this repository you

0 Apr 12, 2022

Rotation-Only Bundle Adjustment

ROBA: Rotation-Only Bundle Adjustment Paper, Video, Poster, Presentation, Supplementary Material In this repository, we provide the implementation of

51 Nov 29, 2022

Code for the paper Open Sesame: Getting Inside BERT's Linguistic Knowledge.

Open Sesame This repository contains the code for the paper Open Sesame: Getting Inside BERT's Linguistic Knowledge. Credits We built the project on t

9 Jul 24, 2022

GestureSSD CBAM - A gesture recognition web system based on SSD and CBAM, using pytorch, flask and node.js

GestureSSD_CBAM A gesture recognition web system based on SSD and CBAM, using pytorch, flask and node.js SSD implementation is based on https://github

2 Jan 06, 2022

(3DV 2021 Oral) Filtering by Cluster Consistency for Large-Scale Multi-Image Matching

Scalable Cluster-Consistency Statistics for Robust Multi-Object Matching (3DV 2021 Oral Presentation) Filtering by Cluster Consistency (FCC) is a very

11 Sep 28, 2022

Code for the paper "Curriculum Dropout", ICCV 2017

Curriculum Dropout Dropout is a very effective way of regularizing neural networks. Stochastically "dropping out" units with a certain probability dis

21 Jan 02, 2022

Jupyter Dock is a set of Jupyter Notebooks for performing molecular docking protocols interactively, as well as visualizing, converting file formats and analyzing the results.

173 Dec 25, 2022

Video Swin Transformer - PyTorch

Video-Swin-Transformer-Pytorch This repo is a simple usage of the official implementation "Video Swin Transformer". Introduction Video Swin Transforme

116 Dec 20, 2022

A Keras implementation of YOLOv3 (Tensorflow backend)

keras-yolo3 Introduction A Keras implementation of YOLOv3 (Tensorflow backend) inspired by allanzelener/YAD2K. Quick Start Download YOLOv3 weights fro

7.1k Jan 03, 2023

NeoDTI: Neural integration of neighbor information from a heterogeneous network for discovering new drug-target interactions

NeoDTI NeoDTI: Neural integration of neighbor information from a heterogeneous network for discovering new drug-target interactions (Bioinformatics).

62 Nov 26, 2022

Deep Learning Tutorial for Kaggle Ultrasound Nerve Segmentation competition, using Keras

Deep Learning Tutorial for Kaggle Ultrasound Nerve Segmentation competition, using Keras This tutorial shows how to use Keras library to build deep ne

922 Dec 19, 2022

PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

11 Jul 28, 2022

A machine learning malware analysis framework for Android apps.

🕵️ A machine learning malware analysis framework for Android apps. ☢️ DroidDetective is a Python tool for analysing Android applications (APKs) for p

77 Dec 27, 2022

Computer Vision Paper Reviews with Key Summary of paper, End to End Code Practice and Jupyter Notebook converted papers

Computer-Vision-Paper-Reviews Computer Vision Paper Reviews with Key Summary along Papers & Codes. Jonathan Choi 2021 The repository provides 100+ Pap

2 Mar 17, 2022

Official Implementation of SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Official Implementation of SimIPU SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations Since

37 Dec 01, 2022