Whole Body Motion Tracking

This repository contains the training, evaluation, and deployment assets for a whole-body motion tracking policy built on top of the GentleHumanoid codebase.

The main focus of this repository is:

• training a universal, robust, and highly dynamic whole-body motion tracking policy,
• supporting upper-body compliance-aware behavior for contact-rich interaction,
• supporting robust live VR teleoperation through a separate teleop stack.

The simulation and training backend is based on mjlab.

A demo of the pretrained policy, showing one model generalizing across diverse and highly dynamic motions, is available here.

teleop_demo.mp4

motion_tracking_demo.mp4

Instructions for deployment and runtime usage are available in the sim2real folder.

Installation

If you do not have uv installed, you can install it following the instructions in the uv documentation.

# this project uses uv for dependency and environment management
uv sync

Motion Dataset Preparation

Quick Start: Download Preprocessed Dataset (Google Drive)

Download link:

• Google Drive Dataset

After downloading, extract it into the repository dataset/ directory, and you should have the following structure:

dataset/
  amass_all/
    meta.json
    ...
  lafan_all/
    ...

Build Dataset from AMASS/LAFAN with GMR

Retargeting with GMR

We use GMR to retarget the AMASS and LAFAN datasets. The output format is a dataset containing a series of npz files with the following fields:

• fps: Frame rate
• root_pos: Root position
• root_rot: Root rotation in quaternion format (xyzw)
• dof_pos: Degrees of freedom positions
• local_body_pos: Local body positions
• local_body_rot: Local body rotations
• body_names: List of body names
• joint_names: List of joint names

You can use the modified version of GMR to directly export npz files that meet the requirements.

You should organize the processed datasets in the following structure:

<dataset_root>/
    AMASS/ACCAD/Female1General_c3d/A1_-_Stand_stageii.npz
    ...
    LAFAN/walk1_subject1.npz
    ...

Dataset Building

Modify DATASET_ROOT in generate_dataset.sh to point to your dataset root directory, then run the script to generate the dataset:

bash generate_dataset.sh

The dataset will be generated in the dataset/ directory, and the code will automatically load these datasets. You can also use the MEMATH environment variable to specify the dataset root path.

Training

You can use the provided train.sh script to run the full training pipeline. Modify the global configuration section in train.sh to set your WandB account and other parameters, then run:

bash train.sh

Under standard settings, training takes approximately 15 hours on 4× A100 GPUs. If GPU memory is constrained, it is recommended to appropriately tune the NPROC and num_envs parameters in train.sh and cfg/task/G1/G1.yaml, respectively. Such adjustments may increase training time and could affect training performance to some extent.

Evaluation

uv run scripts/eval.py --run_path ${wandb_run_path} -p # p for play
uv run scripts/eval.py --run_path ${wandb_run_path} -p --export # export the policy to onnx (sim2real)

If you export a deployment policy, the exported checkpoint will be written under scripts/exports/<task-name>-<timestamp>/.

To use it in the deployment runtime:

1. Copy the exported policy folder (including policy.onnx, policy.pt, and policy.json) into sim2real/assets/ckpts/.
2. Update sim2real/config/tracking.yaml so that policy_path points to the new ONNX file.

For the actual deployment-side test procedure:

• see sim2real/README.md for sim2sim/sim2real testing

That README also explains how to use the UDP motion selector and the VR motion source during deployment.