Maze Runner

A reinforcement learning project where an agent learns to navigate a 2D maze to reach a goal using Q-learning.

Project Structure

• nn.py: Defines the neural network architecture. The Agent class is a feed-forward neural network that takes the current state (position, goal, and surrounding view) as input and predicts the Q-values for four possible actions (Up, Down, Left, Right).
• main.py: A FastAPI server that acts as the environment and training coordinator. It generates the maze, manages the agent's current position, implements the Q-learning update logic, and provides endpoints for the frontend to request actions and save the model.
• main.html: The basic HTML structure that provides a container for the 2D maze visualization.
• main.js: Handles the frontend logic. It fetches the maze layout and start/goal positions from the server, renders the grid on the canvas, and continuously calls the server to move the agent and update the visualization until the goal is reached.

Setup Instructions

Prerequisites

• Python 3.x
• PyTorch
• FastAPI
• Uvicorn (for running the FastAPI server)
• NumPy

Installation

1. Install the required Python packages:

   pip install torch fastapi uvicorn numpy

2. Note on Hardware Acceleration: The current code uses device = 'mps' for Apple Silicon (Metal Performance Shaders). If you are using a CUDA-enabled GPU or a CPU, change the device variable in nn.py and main.py to 'cuda' or 'cpu'.

Running the Project

1. Start the backend server:

   uvicorn main:app --reload

2. Open main.html in any modern web browser.

How It Works

The Environment

• The project generates a 10x10 grid with randomly placed obstacles.
• A start position is chosen from the bottom row, and a goal position is determined using Breadth-First Search (BFS) to find the farthest reachable cell from the start.

The Agent

• State: The input to the network consists of the agent's coordinates, the goal's coordinates, and a local 4-direction view (indicating if there is a wall in each direction).
• Actions: The agent can move in four directions: Up, Down, Left, or Right.
• Training: The agent uses an epsilon-greedy strategy to explore the maze. It learns by updating its Q-values using the Mean Squared Error (MSE) between the current Q-prediction and the target (reward + discounted max Q-value of the next state).
• Rewards:
  • Reaching the goal: +100
  • Hitting a wall/obstacle: -1
  • Moving to an empty cell: +1

Workflow

1. The frontend requests the maze layout from /data.
2. The frontend enters a loop, calling /agent to get the next action and new position.
3. The server performs a training step, updates the agent's weights, and returns the new position.
4. Once the agent reaches the goal, the frontend calls /save to persist the model weights to agent.pt.