|
1 | 1 | # MXCP Project Deployment Template
|
2 | 2 |
|
3 |
| -This repository provides standardized deployment infrastructure for MXCP projects, enabling consistent CI/CD and deployment patterns across all RAW Labs MXCP implementations. |
| 3 | +> 🚀 **Standardized deployment infrastructure for MXCP projects with proven CI/CD patterns** |
| 4 | +
|
| 5 | +This template enables consistent deployment of MXCP servers across RAW Labs and external teams, with support for AWS App Runner and Kubernetes/Flux deployments. |
| 6 | + |
| 7 | +## 🚀 Quick Start |
| 8 | + |
| 9 | +```bash |
| 10 | +# For new projects |
| 11 | +cp -r mxcp-project-deployment-template/ my-new-project/ |
| 12 | +cd my-new-project |
| 13 | +./setup-project.sh my-project-name |
| 14 | + |
| 15 | +# For existing projects |
| 16 | +cd existing-project |
| 17 | +cp -r /path/to/template/.github . |
| 18 | +cp -r /path/to/template/deployment . |
| 19 | +./setup-project.sh my-project-name |
| 20 | +``` |
| 21 | + |
| 22 | +## Table of Contents |
| 23 | + |
| 24 | +- [Quick Start](#-quick-start) |
| 25 | +- [Architecture Overview](#architecture-overview) |
| 26 | +- [Purpose](#purpose) |
| 27 | +- [Template Components](#template-components) |
| 28 | +- [Prerequisites](#prerequisites) |
| 29 | +- [Usage](#usage) |
| 30 | + - [For New MXCP Projects](#for-new-mxcp-projects) |
| 31 | + - [For Existing MXCP Projects](#for-existing-mxcp-projects) |
| 32 | +- [Template Philosophy](#template-philosophy) |
| 33 | +- [Justfile Guide](#justfile-guide) |
| 34 | + - [3-Tiered Testing Architecture](#3-tiered-testing-architecture) |
| 35 | + - [Template Placeholders](#template-placeholders) |
| 36 | + - [Available Tasks](#available-tasks) |
| 37 | +- [Examples](#examples) |
| 38 | +- [Integration Guide for DevOps Teams](#integration-guide-for-devops-teams) |
| 39 | + - [For RAW Labs Teams](#for-raw-labs-teams) |
| 40 | + - [For External Teams (Squirro)](#for-external-teams-squirro) |
| 41 | + - [Network and Service Discovery](#network-and-service-discovery) |
| 42 | + - [Environment Variables](#environment-variables) |
| 43 | + - [Production Checklist](#production-checklist) |
| 44 | +- [Secret Management](#secret-management) |
| 45 | +- [Monitoring and Observability](#monitoring-and-observability) |
| 46 | + - [Health Monitoring](#health-monitoring) |
| 47 | + - [Audit Logs](#audit-logs) |
| 48 | + - [Metrics and Dashboards](#metrics-and-dashboards) |
| 49 | +- [Backup and Recovery](#backup-and-recovery) |
| 50 | +- [Support](#support) |
4 | 51 |
|
5 | 52 | ## Architecture Overview
|
6 | 53 |
|
@@ -119,48 +166,80 @@ The diagram above shows how the template's components work together:
|
119 | 166 |
|
120 | 167 | ## Purpose
|
121 | 168 |
|
122 |
| -**Template Architecture Benefits:** |
123 |
| -- **Standardized CI/CD** - Same deployment logic across all MXCP projects |
124 |
| -- **Easy Squirro collaboration** - Clear separation between stable templates and customizable configuration |
125 |
| -- **Proven patterns** - Based on successful production deployments |
126 |
| -- **Minimal merge conflicts** - Template files rarely change |
| 169 | +### Why Use This Template? |
| 170 | + |
| 171 | +| Benefit | Description | |
| 172 | +|---------|-------------| |
| 173 | +| **🔧 Standardized CI/CD** | Same deployment logic across all MXCP projects | |
| 174 | +| **🤝 Multi-team Support** | Works for RAW Labs, Squirro, and other external teams | |
| 175 | +| **✅ Production Proven** | Powers UAE MXCP Server with 3M+ records | |
| 176 | +| **🔄 Easy Updates** | Minimal merge conflicts, clear separation of concerns | |
| 177 | +| **🏃 Fast Deployment** | From zero to deployed in under 10 minutes | |
| 178 | +| **🔒 Security First** | Built-in secrets management and audit logging | |
127 | 179 |
|
128 | 180 | ## Template Components
|
129 | 181 |
|
130 |
| -### Stable Components (.github/) |
131 |
| -**Never modified by project teams:** |
132 |
| -- `workflows/deploy.yml` - Generic CI/CD pipeline for AWS App Runner |
133 |
| -- `workflows/test.yml` - Standardized testing workflow |
134 |
| -- `scripts/deploy-app-runner.sh` - App Runner deployment logic |
135 |
| -- `workflows/release.yml` - Release management |
136 |
| - |
137 |
| -### Customizable Components (deployment/) |
138 |
| -**Modified for each project:** |
139 |
| -- `config.env.template` - AWS account, ECR repository, service names |
140 |
| -- `mxcp-site-docker.yml.template` - MXCP configuration with project name |
141 |
| -- `profiles-docker.yml.template` - dbt profiles with project name |
142 |
| -- `mxcp-user-config.yml.template` - MXCP user config with LLM API keys and generic secrets |
143 |
| -- `Dockerfile` - Generic container build pattern |
144 |
| -- `start.sh` - Generic container startup script |
145 |
| -- `requirements.txt` - Base MXCP dependencies |
| 182 | +### 📁 Directory Structure |
| 183 | + |
| 184 | +``` |
| 185 | +mxcp-project-deployment-template/ |
| 186 | +├── .github/ # Stable CI/CD (rarely modified) |
| 187 | +│ ├── workflows/ |
| 188 | +│ │ ├── deploy.yml # Main deployment pipeline |
| 189 | +│ │ ├── test.yml # PR testing workflow |
| 190 | +│ │ └── release.yml # Release management |
| 191 | +│ └── scripts/ |
| 192 | +│ └── deploy-app-runner.sh # AWS deployment script |
| 193 | +├── deployment/ # Customizable configs |
| 194 | +│ ├── config.env.template # AWS settings |
| 195 | +│ ├── Dockerfile # Container build |
| 196 | +│ ├── mxcp-site-docker.yml.template # MXCP config |
| 197 | +│ ├── mxcp-user-config.yml.template # Secrets config |
| 198 | +│ ├── profiles-docker.yml.template # dbt profiles |
| 199 | +│ ├── requirements.txt # Python dependencies |
| 200 | +│ └── start.sh # Container startup |
| 201 | +├── .squirro/ # External team integration |
| 202 | +│ ├── setup-for-squirro.sh.template |
| 203 | +│ └── merge-from-raw.sh |
| 204 | +├── justfile.template # Task runner config |
| 205 | +├── setup-project.sh # One-click setup |
| 206 | +├── ENVIRONMENT.md.template # Variable documentation |
| 207 | +└── README.md # This file |
| 208 | +``` |
146 | 209 |
|
147 | 210 | ## Prerequisites
|
148 | 211 |
|
149 |
| -### AWS Setup |
150 |
| -1. **AWS Account** with App Runner service access |
151 |
| -2. **IAM Role**: `AppRunnerECRAccessRole` with ECR access permissions |
152 |
| -3. **Repository Secrets** in GitHub: |
153 |
| - - `AWS_ACCESS_KEY_ID` - For deployment access |
154 |
| - - `AWS_SECRET_ACCESS_KEY` - For deployment access |
155 |
| - - `MXCP_DATA_ACCESS_KEY_ID` - For data download (if needed) |
156 |
| - - `MXCP_DATA_SECRET_ACCESS_KEY` - For data download (if needed) |
157 |
| - - `OPENAI_API_KEY` - For LLM functionality (optional) |
158 |
| - - `ANTHROPIC_API_KEY` - For Claude models (optional) |
159 |
| - |
160 |
| -### Required Tools |
161 |
| -- **Git** - For repository management |
162 |
| -- **Docker** - For local testing (optional) |
163 |
| -- **just** - Modern task runner (optional but recommended) |
| 212 | +### ✅ Required |
| 213 | + |
| 214 | +| Component | Purpose | Setup Guide | |
| 215 | +|-----------|---------|-------------| |
| 216 | +| **AWS Account** | Deployment target | [AWS Free Tier](https://aws.amazon.com/free/) | |
| 217 | +| **GitHub Account** | CI/CD and version control | [GitHub Signup](https://github.com/join) | |
| 218 | +| **IAM Role** | `AppRunnerECRAccessRole` | See ENVIRONMENT.md | |
| 219 | + |
| 220 | +### 🔑 GitHub Secrets Required |
| 221 | + |
| 222 | +```bash |
| 223 | +# Deployment credentials |
| 224 | +gh secret set AWS_ACCESS_KEY_ID |
| 225 | +gh secret set AWS_SECRET_ACCESS_KEY |
| 226 | + |
| 227 | +# Data access (if using S3/external data) |
| 228 | +gh secret set MXCP_DATA_ACCESS_KEY_ID |
| 229 | +gh secret set MXCP_DATA_SECRET_ACCESS_KEY |
| 230 | + |
| 231 | +# LLM APIs (if using AI features) |
| 232 | +gh secret set OPENAI_API_KEY # Optional |
| 233 | +gh secret set ANTHROPIC_API_KEY # Optional |
| 234 | +``` |
| 235 | + |
| 236 | +### 🛠️ Local Development Tools |
| 237 | + |
| 238 | +| Tool | Required | Installation | |
| 239 | +|------|----------|--------------| |
| 240 | +| Git | ✅ Yes | `apt install git` | |
| 241 | +| Docker | ⚠️ Optional | [Docker Desktop](https://docker.com) | |
| 242 | +| just | ⚠️ Recommended | `curl -sSf https://just.systems/install.sh \| bash` | |
164 | 243 |
|
165 | 244 | ## Usage
|
166 | 245 |
|
@@ -490,17 +569,16 @@ just ci-tests-with-data # Standard CI tests with data
|
490 | 569 |
|
491 | 570 | ## Examples
|
492 | 571 |
|
493 |
| -### Successful Implementation |
494 |
| -- **UAE Business Licenses**: https://github.com/raw-labs/uae-mxcp-server |
495 |
| -- **Live Service**: https://sqt3yghjpw.eu-west-1.awsapprunner.com |
496 |
| -- **Records**: 3,186,320 UAE business licenses |
497 |
| -- **Configuration**: 4 vCPU, 8GB memory |
498 |
| - |
499 |
| -### Template Benefits Proven |
500 |
| -- **Zero merge conflicts** during RAW-Squirro collaboration |
501 |
| -- **Consistent deployment** across all MXCP projects |
502 |
| -- **Easy customization** - Only edit deployment/config.env |
503 |
| -- **Automatic CI/CD** - Works out of the box |
| 572 | +### 🏆 Success Story: UAE Business Licenses |
| 573 | + |
| 574 | +| Metric | Value | |
| 575 | +|--------|-------| |
| 576 | +| **Repository** | [uae-mxcp-server](https://github.com/raw-labs/uae-mxcp-server) | |
| 577 | +| **Live Service** | [App Runner Deployment](https://sqt3yghjpw.eu-west-1.awsapprunner.com) | |
| 578 | +| **Data Scale** | 3,186,320 business licenses | |
| 579 | +| **Performance** | 4 vCPU, 8GB RAM | |
| 580 | +| **Deployment Time** | < 10 minutes | |
| 581 | +| **Merge Conflicts** | Zero during RAW-Squirro collaboration | |
504 | 582 |
|
505 | 583 | ## Integration Guide for DevOps Teams
|
506 | 584 |
|
|
0 commit comments