Event-based Semantic-aided Motion Segmentation

A multi-task neural network that processes event data in order to jointly output multi-class semantic segmentation and motion segmentation masks.

This work reaches state-of-the-art results on EVIMO and EVIMO2 datasets.

Written with pytorch, from the previous work of https://prg.cs.umd.edu/EV-IMO.html.

Datasets

This work has been done using EVIMO and EVIMO2 datasets, from https://better-flow.github.io/evimo/.

To run our network, we needed to process the datasets following these instructions:

• For the dataset EVIMO, use the code here to regenerate the depth image sequences with background.

• For the dataset EVIMO2, use our code in generate_evimo to generate the ground truth IMO mask from objects velocities.

Fusion of both datasets together is not possible as data formatting and semantic classes are too different.

Usage

• For training

    data_dir=/path/to/dataset/

    CUDA_VISIBLE_DEVICES=0 python3 main.py $data_dir -j 8 --batch-size 32 -f 50 --lr 1e-2 --sequence-length 5 --log-output --with-gt --epochs 50 -c 4 >evimo1.log&

• For testing

    dispnet_dir=/path/to/dispnet_model_best.pth.tar

    posenet_dir=/path/to/exp_pose_model_best.pth.tar

    data_dir=/path/to/test/sequence/

    output_dir=/path/to/save

    python3 run_inference.py --dataset-dir $data_dir --pretrained-dispnet $dispnet_dir --pretrained-posenet $posenet_dir --sequence-length 5 --norm-type fd --output-dir $output_dir -c 4

There are some subtle differences between the training and testing codes of EVIMO and EVIMO2. Please see the code comments for details. With EVIMO2 initial semantic classes are numerous with very few samples, the semantic segmentation class restructuring is done in the code.

Citation

When using this code, please cite our work:

@inproceedings{jiang:visapp2024,
  author = {Jiang, Chenao and Moreau, Julien and Davoine, Franck},
  title = {Event-Based Semantic-Aided Motion Segmentation},
  booktitle = {19th International Conference on Computer Vision Theory and Applications ({VISAPP})},
  volume = {4},
  pages = {159--171},
  publisher = {{SCITEPRESS}},
  year = {2024},
  month = Feb,
  address = {Rome, Italy}
}

License

Licensed under the EUPL-1.2-or-later