Reinforcement Learning Project

This project was created to make it easier to get started with Reinforcement Learning. It now contains:

• An implementation of the DDPG Algorithm in Python, which works for both single-agent environments and multi-agent environments.
• Single and parallel environments in Unity ML agents using the Python API.
• Two Jupyter notebooks:
  • 3DBall.ipynb: This is a simple example to get started with Unity ML Agents & the DDPG Algorithm.
  • 3DBall_parallel_environment.ipynb: The same, but now for an environment run in parallel.

Getting Started

Install Basic Dependencies

To set up your python environment to run the code in the notebooks, follow the instructions below.

• If you're on Windows I recommend installing Miniforge. It's a minimal installer for Conda. I also recommend using the Mamba package manager instead of Conda. It works almost the same as Conda, but only faster. There's a cheatsheet of Conda commands which also work in Mamba. To install Mamba, use this command:

conda install mamba -n base -c conda-forge 

• Create (and activate) a new environment with Python 3.6 or later. I recommend using Python 3.9:

  • Linux or Mac:

  mamba create --name rl39 python=3.9 numpy
  source activate rl39

  • Windows:

  mamba create --name rl39 python=3.9 numpy
  activate rl39

• Install PyTorch by following instructions on Pytorch.org. For example, to install PyTorch on Windows with GPU support, use this command:

mamba install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

• Install additional packages:

mamba install jupyter notebook matplotlib

• Create an IPython kernel for the rl39 environment in Jupyter.

python -m ipykernel install --user --name rl39 --display-name "rl39"

• Change the kernel to match the rl39 environment by using the drop-down menu Kernel -> Change kernel inside Jupyter Notebook.

Install Unity Machine Learning Agents

Note: In order to run the notebooks on Windows, it's not necessary to install the Unity Editor, because I have provided the standalone executables of the environments for you.

Unity ML Agents is the software that we use for the environments. The agents that we create in Python can interact with these environments. Unity ML Agents consists of several parts:

• The Unity Editor is used for creating environments. To install:

  • Install Unity Hub.
  • Install the latest version of Unity by clicking on the green button Unity Hub on the download page.

  To start the Unity editor you must first have a project:

  • Start the Unity Hub.
  • Click on "Projects"
  • Create a new dummy project.
  • Click on the project you've just added in the Unity Hub. The Unity Editor should start now.

• The Unity ML-Agents Toolkit. Download the latest release of the source code or use the Git command: git clone --branch release_18 https://github.com/Unity-Technologies/ml-agents.git.

• The Unity ML Agents package is used inside the Unity Editor. Please read the instructions for installation.

• The mlagents Python package is used as a bridge between Python and the Unity editor (or standalone executable). To install, use this command: python -m pip install mlagents==0.27.0. Please note that there's no conda package available for this.

Install an IDE for Python

For Windows, I would recommend using PyCharm (my choice), or Visual Studio Code. Inside those IDEs you can use the Conda environment you have just created.

Creating a custom Unity executable

Load the examples project

The Unity ML-Agents Toolkit contains several example environments. Here we will load them all inside the Unity editor:

• Start the Unity Hub.
• Click on "Projects"
• Add a project by navigating to the Project folder inside the toolkit.
• Click on the project you've just added in the Unity Hub. The Unity Editor should start now.

Create a 3D Ball executable

The 3D Ball example contains 12 environments in one, but this doesn't work very well in the Python API. The main problem is that there's no way to reset each environment individually. Therefore, we will remove the other 11 environments in the editor:

• Load the 3D Ball scene, by going to the project window and navigating to Examples -> 3DBall -> Scenes-> 3DBall
• In the Hierarchy window select the other 11 3DBall objects and delete them, so that only the 3DBall object remains.

Next, we will build the executable:

• Go to File -> Build Settings
• In the Build Settings window, click Build
• Navigate to notebooks folder and add 3DBall to the folder name that is used for the build.

Instructions for running the notebooks

1. Download the Unity executables for Windows. In case you're not on Windows, you have to build the executables yourself by following the instructions above.
2. Place the Unity executable folders in the same folder as the notebooks.
3. Load a notebook with Jupyter notebook. (The command to start Jupyter notebook is jupyter notebook)
4. Follow further instructions in the notebook.