ROS2 Jazzy + Gazebo Harmonic + PX4 + ROSA + ChatOllama Docker Environment

A complete Docker-based environment for autonomous robotics featuring ROS2 Jazzy, Gazebo Harmonic simulation, PX4 autopilot integration, NASA's ROSA task planning framework, and AI capabilities with Ollama/LangChain.

📋 Table of Contents

• Demo
• Future Vision
• Features
• Prerequisites
• Installation
• Setting Up the ROS2 Agent and Simulation
• Usage
• Graphics Diagnostics
• Directory Structure
• Container Management
• Acknowledgments
• License
• Contact
• Additional Resources

🖼️ Demo

Demo of drone simulation with ROS2 Agent and Ollama integration

Sim TF-Tree

The TF Tree for the simulation for the Go2 and UAV robots

Gazebo Environment

Gazebo for the two robots

Rviz2

Rivz2 for the two robots

🔮 Future Vision

Multi-Robot ROS2 Agent System Architecture - End Goal

This represents our ultimate vision for a comprehensive multi-robot coordination system. The current implementation focuses on drone control, while this architecture showcases the planned expansion to include wheeled robots, legged robots (Unitree Go2), and advanced mission coordination capabilities.

System Flow Architecture:

🔄 Information Flow:

1. User → Multi-Robot CLI → ROSA Agent (Natural Language Commands)
2. ROSA Agent → Robot-Specific Tools (Decision Making & Tool Selection)
3. Robot Tools → ROS2 Topics & Clients (Hardware Communication)
4. Topics & Clients → CLI → User (Feedback & Status Loop)

📡 Communication Layers:

• MAVROS Topics & Clients: Drone communication (state, pose, gimbal, setpoints)
• Nav Topics & Clients: Wheeled robot communication (cmd_vel, odometry, mapping, scanning)
• Go2 Topics & Clients: Legged robot communication (joint_states, walking commands, IMU, footstep planning)

🧠 Central Intelligence: The ROSA Agent serves as the central decision-making brain, utilizing:

• System Prompts: Robot coordination, safety guidelines, mission context
• Configuration System: robots.yaml defining robot capabilities and topic mappings
• Multi-Agent LLM: Enhanced AI coordination between multiple robots

📈 Key Features:

• Bidirectional Communication: Real-time feedback from robots through CLI interface
• Multi-Robot Coordination: Simultaneous control of different robot types
• Natural Language Interface: Intuitive command structure for complex missions
• Scalable Architecture: Easy addition of new robot types and capabilities

Planned System Components:

• Multi-Robot CLI: Unified command interface for all robot types with real-time status display
• Robot Manager: Lifecycle management and resource allocation across robot fleet
• Mission Coordinator: Task allocation and multi-robot synchronization for complex operations
• Robot Factory: Dynamic robot instance creation and management
• Multi-Agent LLM: Enhanced AI coordination between multiple robots with shared situational awareness
• Configuration Management: Dynamic robot discovery and capability mapping through robots.yaml

🚀 Features

Core Robotics Stack (FIXED VERSION)

• ROS2 Jazzy Desktop - Latest Robot Operating System 2 with complete desktop features
• Gazebo Harmonic - Modern 3D robot simulation with Qt6 and enhanced graphics support
• XRCE-DDS Agent - Lightweight DDS middleware for embedded systems
• MAVROS - MAVLink communication bridge for PX4/ArduPilot
• rqt_tf - ROS Transform visualization and debugging

Flight Systems

• PX4 Development Tools - Complete toolchain for PX4 autopilot development
• JSBSim - Flight dynamics and control simulator
• ros_gz_bridge - Custom-built bridge between ROS2 and Gazebo
• SITL (Software-in-the-loop) - PX4 simulation environment

AI & Task Planning

• ROSA (NASA JPL) - ROS Agent task planning framework
• Ollama + ChatOllama - Local LLM integration with Qwen3:8b model
• LangChain - AI application development framework

Development Tools

• VS Code - Complete IDE integrated in container
• RQt tools - Robotics visualization and debugging with Qt6 support
• Python Development Environment - Complete toolchain with pip packages
• gedit - Text editor for quick edits

FIXED VERSION Graphics Stack

• Qt6 Support - Full Qt6 platform with XCB backend
• Mesa Graphics - Ubuntu 24.04 compatible OpenGL libraries
• Graphics Diagnostics - Built-in tools for troubleshooting GUI issues
• Fallback Rendering - Automatic hardware → software rendering cascade
• X11 Auto-Detection - Smart display detection with multiple fallbacks

📋 Prerequisites

• Ubuntu 22.04 or 24.04 (host system) - Optimized for Ubuntu 24.04
• Docker installed (version 19.03+)
• NVIDIA GPU support (optional, software fallbacks included)
• X11 server for GUI applications
• 20GB+ free disk space

🔧 Installation

1. Clone the Repository

git clone https://github.com/AbdullahGM1/ros2_agent_sim_docker.git
cd ros2-agent-sim-docker

2. Make the Script Executable

chmod +x docker_run.sh

3. Run the Environment (FIXED VERSION)

./docker_run.sh

Note(1): The script will automatically check if the Docker image exists and build it if necessary. The building process may take 30-60 minutes depending on your system specifications and internet connection. This FIXED VERSION includes comprehensive Ubuntu 24.04 compatibility and graphics environment setup.

Note(2): To install the LLM Model, you need to uncomment the section in Dockerfile.ros2-agent-sim file to pull the LLM PHASE 14.5. Or, you can do it manually inside the container:

ollama pull qwen3:8b 
ollama pull qwen2.5vl:7b 

The automated process includes:

• Docker image building with Ubuntu 24.04 fixes (if not exists)
• ROS2 Jazzy installation
• Gazebo Harmonic setup with Qt6 support
• PX4 development environment
• Ollama and Qwen3:8b model download
• All necessary dependencies with graphics fixes

4. Install Dependencies Inside the Container

Once inside the container, run the installation script to set up all dependencies:

cd /home/user/shared_volume
./install.sh

The install script includes:

• Ubuntu 24.04 package verification
• Graphics environment validation
• Enhanced error handling and diagnostics
• Comprehensive workspace setup

⚠️ ATTENTION ⚠️

If the ros2_agent_sim package was not automatically cloned to ros2_ws/src/ during installation, you must manually clone it using the commands below.

🔄 Setting Up the ROS2 Agent and Simulation

After completing the installation steps above, follow these steps to set up the ROS2 Agent and simulation environment:

1. Clone the ROS2 Agent Simulation Package

cd ~/ros2_ws/src/
git clone --recursive https://github.com/AbdullahGM1/ros2_agent_sim.git

This package (ros2_agent_sim) contains:

• ROS2 Agent Package - For LLM-based robot interaction
• Simulation environment - Integrated with PX4 for drone simulation

2. Build the Package

cd ~/ros2_ws
colcon build 

3. Source the Setup Files

source install/setup.bash

4. Launch the SAR Simulation (Enhanced Graphics Support)

ros2 launch sar_system sar_system.launch.py

This will launch a drone simulation that is connected to PX4 autopilot with Qt6 and enhanced graphics support.

5. Run the ROS2 Agent

In a new terminal (inside the container), run:

source ~/ros2_ws/install/setup.bash
ros2 run ros2_agent ros2_agent_node

This launches the interactive CLI interface to communicate with and control the robots.

🔨 Usage

Interacting with the Drone

The ROS2 Agent provides a natural language interface to command the drone. Example commands:

> Take off to 2 meters
> Fly to position x=10, y=5, z=3
> Land
> What is your position
> Show me the camera feed

📁 Directory Structure

ros2-agent-sim-docker/
├── docker_run.sh              # Enhanced script with graphics support (build + run)
├── docker/
│   └── Dockerfile.ros2-agent-sim  # FIXED Dockerfile with Ubuntu 24.04 support
├── middleware_profiles        # DDS configuration profiles
│   └── rtps_udp_profile.xml
├── PX4_config                 # PX4 configuration files
│   ├── px4/
│   │   ├── 4020_gz_x500_d435
│   │   ├── 4021_gz_x500_lidar_camera
│   │   ├── 4022_gz_x3_uav
│   │   └── CMakeLists.txt
│   └── worlds/                # Simulation worlds
│       └── default.sdf
├── README.md
└── scripts/                   # Enhanced container scripts with graphics support
    ├── entrypoint.sh          # FIXED entrypoint with Ubuntu 24.04 compatibility
    ├── install.sh             # Enhanced installation with graphics validation
    ├── bashrc_template.sh     # Enhanced bashrc with graphics environment
    └── requirements.txt

🐳 Container Management

Container Access and Credentials

• Default password for the user in the container: user

Starting the Container (Enhanced)

# Simple startup with graphics support (automatic build if needed)
./docker_run.sh

# Check container and graphics status
./docker_run.sh status

# Force rebuild with Ubuntu 24.04 fixes
./docker_run.sh rebuild

Enhanced Container Management Commands

# Stop container
docker stop ros2_agent_sim

# Remove container
docker rm ros2_agent_sim

# Remove image (full cleanup)
docker rmi ros2-agent-sim:latest

# Check logs
docker logs ros2_agent_sim

# Force rebuild from scratch with fixes
./docker_run.sh rebuild

# Open additional shell
./docker_run.sh shell

# Show comprehensive status
./docker_run.sh status

Container Graphics Environment

The FIXED VERSION automatically configures:

• X11 Authentication: Multi-display detection and fallbacks
• Qt6 Environment: Platform plugins and conflict resolution
• OpenGL Support: Hardware acceleration with software fallbacks
• GPU Passthrough: NVIDIA/AMD/Intel GPU support
• Graphics Debugging: Built-in diagnostic tools

🎉 Acknowledgments

This project builds upon the excellent work of:

• ROSA (NASA JPL) - ROS Agent task planning framework
• smart_track Docker Environment by Mohammed Abdelkader

Special thanks to Mohammed Abdelkader for providing the foundational Docker configurations and ROS2-PX4 integration scripts that made this project possible.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Contact

Abdullah GM - @AbdullahGM1 - [email protected]

📚 Additional Resources

• ROS2 Documentation
• Gazebo Harmonic Documentation
• PX4 User Guide
• NASA ROSA Repository
• Ollama Documentation
• LangChain Documentation
• ROS2 Agent Simulation - The simulation and agent package used in this project
• Qt6 Documentation - For Qt6 platform and graphics information
• Ubuntu 24.04 Release Notes - Ubuntu 24.04 compatibility information

---

FIXED VERSION with Ubuntu 24.04 Support & Enhanced Graphics
Made with ❤️ by Abdullah GM