2D Pose Based Action Recognition

This Project aims at realizing real time 2D Pose Based Action Recognition for the following five human actions:

• running
• walking
• walking the dog
• jogging
• bicycling

• use docker for building an automated training pipeline based on the previous work done by Group 1 and the open-source MPII dataset.
• use docker for building an automated real time activity recognition video stream pipeline.
• use trt_pose for extracting the skeletons.
• use ddnet for recognizing the activities.

This work was done by ZhengChen Guan, Karsten Rudolf and Tobias Heilig during the IW276 Autonome Systeme Labor at the Karlsruhe University of Applied Sciences (Hochschule Karlruhe - Technik und Wirtschaft) in WS 2020 / 2021.

Table of Contents

• Requirements
• Running
• TODO

Requirements

• Python 3.6 (or above)
• OpenCV
• TensorFlow
• Scipy
• Docker
• Jetson Nano

Running ...

The demo aka video stream pipeline as well as the training pipeline have both been containerized with docker to provide a clean runtime environment.

Run docker-build.sh to build the docker image.

... the demo

Run docker-run.sh to execute the video stream pipeline.

First argument - path to a video directory on the host.
Second argument - filename of the video to be processed as found in the path specified by the first argument.
Returns - the output video placed in the video directory specified above.

Example

./docker-run.sh /path/to/videos video.mp4

... the training

We did split the training execution pipeline into two separate steps: prepare the training data and execute the training. This was done to keep it open for the user to actually start the training on his own machine or Google Collab for example (where no docker runs are supported - the training-data has to be uploaded to Google Drive in that case and train_3_train_model.py has to be executed directly there).

1. Run docker-prepare-training.sh to prepare the training data.

First argument - path to a video directory on the host where the sample videos will be downloaded to.
Second argument - path to the training-data directory where the sample data will be placed in.
Returns - the sample data.

Example

./docker-prepare-training.sh /path/to/videos /path/to/training-data

2. Run docker-run-training.sh to start the training.

First argument - path to the training-data directory where the sample data can be found.
Second argument - path to the model directory the resulting pre-trained model .pth will be placed in.
Returns - the pre-trained model.

Example

./docker-run-training.sh /path/to/training-data /path/to/model

TODO

Unfortunately, there is a yet unresolved bug when starting the training. Therefore, we were not able to obtain a pre-trained model and actually use ddnet for activity recognition. In the current state of the project the recognized activities drawn onto the frames come from the pre-processed training data samples. To make the project work lively as intended the following problems have to be solved first:

• Fix segmentation fault when starting the training to obtain a model. See the internal Wiki, contact mickael.cormier AT iosb.fraunhofer.de for more information.
• Execute the model in the demo to recognize the activities and draw the actual results onto the frames.

Acknowledgments

This repo is based on

• IWI276/IW276SS20P1
• TRT_Pose
• DD-Net

Thanks to the original authors for their work!

Contact

Please email mickael.cormier AT iosb.fraunhofer.de for further questions.