ConvNeXtPose

Introduction

This repo is official PyTorch implementation of ConvNeXtPose: A Fast Accurate Method for 3D Human Pose Estimation and its AR Fitness Application in Mobile Devices(IEEE Access 2023).

Dependencies

• PyTorch
• CUDA
• cuDNN
• COCO API

This code is tested under Ubuntu 20.04, CUDA 11.6, cuDNN 8.4.1 environment with single NVIDIA RTX 3090Ti GPUs.

Python 3.8.10 version is used for development.

Directory

Root

The ${ROOT} is described as below.

${ROOT}
|-- data
|-- demo
|-- common
|-- main
|-- tool
|-- vis
`-- output

• data contains data loading codes and soft links to images and annotations directories.
• demo contains demo codes.
• common contains kernel codes for 3d multi-person pose estimation system.
• main contains high-level codes for training or testing the network.
• tool contains data pre-processing codes. You don't have to run this code. I provide pre-processed data below.
• vis contains scripts for 3d visualization.
• output contains log, trained models, visualized outputs, and test result.

Data

You need to follow directory structure of the data as below.

${ROOT}
|-- data
|   |-- Human36M
|   |   |-- bbox_root
|   |   |   |-- bbox_root_human36m_output.json
|   |   |-- images
|   |   |-- annotations
|   |-- MPII
|   |   |-- images
|   |   |-- annotations
|   |-- MSCOCO
|   |   |-- bbox_root
|   |   |   |-- bbox_root_coco_output.json
|   |   |-- images
|   |   |   |-- train2017
|   |   |   |-- val2017
|   |   |-- annotations
|   |-- MuCo
|   |   |-- data
|   |   |   |-- augmented_set
|   |   |   |-- unaugmented_set
|   |   |   |-- MuCo-3DHP.json
|   |-- MuPoTS
|   |   |-- bbox_root
|   |   |   |-- bbox_mupots_output.json
|   |   |-- data
|   |   |   |-- MultiPersonTestSet
|   |   |   |-- MuPoTS-3D.json

• Download Human3.6M parsed data [data]
• Download MPII parsed data [images][annotations]
• Download MuCo parsed and composited data [data]
• Download MuPoTS parsed data [images][annotations]
• All annotation files follow MS COCO format.
• If you want to add your own dataset, you have to convert it to MS COCO format.

To download multiple files from Google drive without compressing them, try this. If you have a problem with 'Download limit' problem when tried to download dataset from google drive link, please try this trick.

* Go the shared folder, which contains files you want to copy to your drive  
* Select all the files you want to copy  
* In the upper right corner click on three vertical dots and select “make a copy”  
* Then, the file is copied to your personal google drive account. You can download it from your personal account.  

Output

You need to follow the directory structure of the output folder as below.

${ROOT}
|-- output
|-- |-- log
|-- |-- model_dump
|-- |-- result
`-- |-- vis

• Creating output folder as soft link form is recommended instead of folder form because it would take large storage capacity.
• log folder contains training log file.
• model_dump folder contains saved checkpoints for each epoch.
• result folder contains final estimation files generated in the testing stage.
• vis folder contains visualized results.

3D visualization

• Run $DB_NAME_img_name.py to get image file names in .txt format.
• Place your test result files (preds_2d_kpt_$DB_NAME.mat, preds_3d_kpt_$DB_NAME.mat) in single or multi folder.
• Run draw_3Dpose_$DB_NAME.m

Running ConvNeXtPose

Start

• In the main/config.py, you can change settings of the model including dataset to use, network backbone, and input size and so on.

Train

In the main folder, run

python train.py --gpu 0-1

to train the network on the GPU 0,1.

If you want to continue experiment, run

python train.py --gpu 0-1 --continue

--gpu 0,1 can be used instead of --gpu 0-1.

Test

Place trained model at the output/model_dump/.

In the main folder, run

python test.py --gpu 0-1 --test_epoch 20-21

to test the network on the GPU 0,1 with 20th and 21th epoch trained model. --gpu 0,1 can be used instead of --gpu 0-1.

Results

Here I report the performance of the ConvNeXtPose.

• Download pre-trained models of the ConvNeXtPose in here

Human3.6M dataset using protocol 1

For the evaluation, you can run test.py or there are evaluation codes in Human36M.

Human3.6M dataset using protocol 2

For the evaluation, you can run test.py or there are evaluation codes in Human36M.

Fitness Application

We will upload the Application code soon in the future

Acknowledgement

Part of our code is borrowed from PoseNet. We thank the authors for releasing the codes.

Reference

Data:

• Compatibility for most of the publicly available 2D and 3D, single and multi-person pose estimation datasets including Human3.6M, MPII, MS COCO 2017, MuCo-3DHP and MuPoTS-3D.

  @inproceedings{nguyen2023convnextpose,
  title={ConvNeXtPose: A Fast Accurate Method for 3D Human Pose Estimation and its AR Fitness Application in Mobile Devices},
  author={Nguyen, Hong Son and Kim, MyoungGon and Im, Changbin and Han, Sanghoon and Han, JungHyun},
  journal={IEEE Access},
  year={2023},
  publisher={IEEE}
}