GeRM: A Generalist Robotic Model with Mixture-of-Experts for Quadruped Robot

IROS 2024

📍 This paper was accepted to be presented at IROS 2024 and I had the honor of delivering an oral talk about it in October 2024 in Abu Dhabi. 🎤🌍

Welcome to the official repository of GeRM: A Generalist Robotic Model with Mixture-of-Experts for Quadruped Robot. It is a vision-language-action(VLA) model with a mixture-of-experts(MoE) architecture and trained in a offline reinforcement learning manner. 🤖🐾

This repository contains the complete code for the data processing, training, and testing pipelines of GeRM. Some of the model architecture code is modified from Robotics Transformer by Google Research. 💻✨

We hope this code helps the community in advancing the field of VLAs and robot learning! 🚀

Features

• 🗂️ Data Preprocessing:
  The data preprocessing code is located in data_process.sh.

• 🏋️‍♂️💻 Training Script:
  The training script is located in train_ddp.sh, and it supports multi-GPU training.

• 🏗️ Model Architecture:
  The model architecture code is located in pytorch_robotics_transformer/transformer_network.py.

• 🧪 Core Testing Code:
  The core testing code is located in pytorch_robotics_transformer/transformer_inference.py.

• 🎮 Agent Training Script:
  The agent training code is located in agent_ddp.py.

Future Directions

💡 One promising direction for future work is to extend our code to apply it to robotic arms. 🤖

Environment Configuration and Dataset

🛠️📦 The environment setup in Isaac Gym is relatively complex and requires extensive configuration. We plan to open source the environment setup and dataset code in the future to make it more accessible.

Issues and Contributions

💬 If you have any questions or issues, feel free to leave a message in the Issues section. We'd love to hear your thoughts and feedback!

Acknowledgments

🙏💡 We would like to thank Google Research for their incredible work on the Robotics Transformer, which provided the foundational model architecture for GeRM.

---

Stay tuned for future updates, and happy coding! 🎉

Citation

If you find this work useful, please consider citing the following paper:

@inproceedings{song2024germ,
  title={Germ: A generalist robotic model with mixture-of-experts for quadruped robot},
  author={Song, Wenxuan and Zhao, Han and Ding, Pengxiang and Cui, Can and Lyu, Shangke and Fan, Yaning and Wang, Donglin},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={11879--11886},
  year={2024},
  organization={IEEE}
}