Serving PyTorch Models in C++

• This repository contains various examples to perform inference using PyTorch C++ API.
• Run git clone https://github.com/Wizaron/pytorch-cpp-inference in order to clone this repository.

Environment

1. Dockerfiles can be found at docker directory. There are two dockerfiles; one for cpu and the other for cuda10. In order to build docker image, you should go to docker/cpu or docker/cuda10 directory and run docker build -t <docker-image-name> ..
2. After creation of the docker image, you should create a docker container via docker run -v <directory-that-this-repository-resides>:<target-directory-in-docker-container> -p 8181:8181 -it <docker-image-name> (We will use 8181 to serve our PyTorch C++ model).
3. Inside docker container, go to the directory that this repository resides.
4. Download libtorch from PyTorch Website (CPU : https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.3.1%2Bcpu.zip - CUDA10 : https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.3.1.zip).
5. Unzip libtorch via unzip. This will create libtorch directory that contains torch shared libraries and headers.

Code Structure

• models directory stores PyTorch models.
• libtorch directory stores C++ torch headers and shared libraries to link the model against PyTorch.
• utils directory stores various utility function to perform inference in C++.
• inference-cpp directory stores codes to perform inference.

Exporting PyTorch ScriptModule

• In order to export torch.jit.ScriptModule of ResNet18 to perform C++ inference, go to models/resnet directory and run python3 resnet.py. It will download pretrained ResNet18 model on ImageNet and create models/resnet_model_cpu.pth and (optionally) models/resnet_model_gpu.pth which we will use in C++ inference.

Serving the C++ Model

• We can either serve the model as a single executable or as a web server.

Single Executable

• In order to build a single executable for inference:
  1. Go to inference-cpp/cnn-classification directory.
  2. Run ./build.sh in order to build executable, named as predict.
  3. Run the executable via ./predict <path-to-image> <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}>.
  4. Example: ./predict image.jpeg ../../models/resnet/resnet_model_cpu.pth ../../models/resnet/labels.txt false

Web Server

• In order to build a web server for production:
  1. Go to inference-cpp/cnn-classification/server directory.
  2. Run ./build.sh in order to build web server, named as predict.
  3. Run the binary via ./predict <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}> (It will serve the model on http://localhost:8181/predict).
  4. Example: ./predict ../../../models/resnet/resnet_model_cpu.pth ../../../models/resnet/labels.txt false
  5. In order to make a request, open a new tab and run python test_api.py (It will make a request to localhost:8181/predict).

Acknowledgement

1. pytorch
2. crow
3. tensorflow_cpp_object_detection_web_server