Neurobee

Neurobee is a framework to learn low-level quadrotor control policies using reinforcement learning in a simulation environment and translate the learned policies to the real world with minimal fine-tuning.

Installation

Run the following commands to install all dependencies for Neurobee:

# Create and activate virtual environment
virtualenv env -p python3
source env/bin/activate

cd neurobee/

# Install dependencies
pip install .

Instructions

Training

Each experiment requires a config file that defines all aspects of the experiment. Some example config files have been provided in config directory.

Run the following command to begin training:

python train.py --config config/ppo__crazyflie_baseline.yml

To display all train.py arguments:

python train.py --help

#  --config CONFIG, -c CONFIG
#                        Path to config file to use for training
#  --resume RESUME, -r RESUME
#                        Path to snapshot from where training should be resumed
#  --log_dir LOG_DIR, -l LOG_DIR
#                        Directory to log into. Defaults to '_experiments/config_file_name'
#  --seed SEED, -s SEED  list of seeds to use separated by comma (or a single seed w/o comma). If None seeds
#                        from config will be used
#  --workers WORKERS, -w WORKERS
#                        Number of worker processes to use to sample from environment (default: 6)
#  --snapshot_mode {all,last,gap,none}, -sm {all,last,gap,none}
#                        Snapshot mode
#  --snapshot_gap SNAPSHOT_GAP, -sg SNAPSHOT_GAP
#                        Snapshot gap when mode is gap
#  --plot                Enable plotting
#  --param_name PARAM_NAME, -p PARAM_NAME
#                        task hyperparameter names separated by comma
#  --param_val PARAM_VAL, -pv PARAM_VAL
#                        task hyperparam values. For a single par separated by comma. For adjacent params
#                        separated by double comma. Ex: "-p par1,par2 -pv pv11,pv12,,pv21,pv22" where pv11,pv12
#                        - par values for par1 , pv21,pv22 - par values for par2

---

Visualization

The performance of a trained policy can be visualized using the following command:

python play_policy.py --snapshot_path _experiments/ppo__crazyflie_baseline/seed_001/_1/

To display all play_policy.py arguments:

python play_policy.py --help

#  -p SNAPSHOT_PATH, --snapshot_path SNAPSHOT_PATH
#                        Path to directory containing the policy snapshot
#  -i SNAPSHOT_ITR, --snapshot_itr SNAPSHOT_ITR
#                        Snapshot iteration to load (can be an integer, 'last', or 'first'
#  -e {QuadrotorEnv,PybulletQuadrotorEnv}, --environment {QuadrotorEnv,PybulletQuadrotorEnv}
#                        Environment to test the policy in (loads the training env by default if not specified).
#  -n N_ROLLOUT, --n_rollout N_ROLLOUT
#                        Number of rollouts.
#  -nd, --non_deterministic
#                        Whether to use non-deterministic policy.
#  -phy {pyb,dyn,pyb_gnd,pyb_drag,pyb_dw,pyb_gnd_drag_dw}, --physics {pyb,dyn,pyb_gnd,pyb_drag,pyb_dw,pyb_gnd_drag_dw}
#                        Physics implementation to use in Pybullet.
#  -ns, --normalize_state
#                        Normalize state.
#  -irs, --init_random_state
#                        Randomly sample initial state.
#  -rg, --resample_goal  Randomly sample goal.
#  -sn, --sensor_noise   Add sensor noise.

---

Compile

To compile a trained policy to firmware executable C code, run the following command:

python compile.py --mode 2 --root_dir _experiments/ppo__crazyflie_baseline/seed_001/ --out_dir compiled_models/

To display all compile.py arguments:

python compile.py --help

#  -m MODE, --mode MODE  Select a mode to copy file.
#                        0: a txt file with directories
#                        1: a root where all the experiments are stored and select the best seeds
#                        2: a root directory where all the sub-directories that contain plk file will be copied
#  -r ROOT_DIR, --root_dir ROOT_DIR
#                        Root directory of the experiments
#  -o OUT_DIR, --out_dir OUT_DIR
#                        Directory to save the experiments
#  -t TXT, --txt TXT     Text file that contains all the models

Steps to use the generated source code:

• Clone the modified crazyflie firmware for Neurobee if not already cloned.
• Place the generated neurobee.c file in crazyflie-firmware/src/modules/src/.
• Start the Crazyflie 2.1 in bootloader mode.
• Build and flash the updated crazyflie firmware.
• Watch in astonishment as your drone flies using a brain of its own!

---

References

[1] Molchanov, Artem, et al. "Sim-to-(multi)-real: Transfer of low-level robust control policies to multiple quadrotors." arXiv preprint arXiv:1903.04628 (2019).

@article{molchanov2019sim,
  title={Sim-to-(multi)-real: Transfer of low-level robust control policies to multiple quadrotors},
  author={Molchanov, Artem and Chen, Tao and H{\"o}nig, Wolfgang and Preiss, James A and Ayanian, Nora and Sukhatme, Gaurav S},
  journal={arXiv preprint arXiv:1903.04628},
  year={2019}
}