Polygon Validator Infrastructure

A comprehensive AWS infrastructure setup for Polygon blockchain nodes using Terraform, with working P2P connections, RPC fixes, and complete automation.

🎯 Project Overview

This project demonstrates building production-ready Polygon validator infrastructure, learning the technical components and operational challenges of running blockchain infrastructure. All sync issues have been solved with proper RPC endpoints, state sync configuration, and working peer connections.

🏗️ What This Infrastructure Builds

Complete Polygon PoS Node Infrastructure supporting:

• ✅ Full Node Operations (current deployment)
• ✅ Data Node Services (RPC endpoints for dApps)
• ✅ Validator-Ready Infrastructure (add staking to become validator)

Current Deployment: Full Node/Data Node

This setup creates a non-validating full node that:

• ✅ Syncs complete blockchain data (Heimdall + Bor)
• ✅ Provides RPC services to applications
• ✅ Supports network decentralization
• ✅ Can be upgraded to validator by adding economic stake

✅ Current Status - FULLY WORKING

• Complete AWS infrastructure deployed with Terraform automation
• Amazon Linux 2023 with proper security configuration
• SSH access with generated key pairs and security groups
• Built Bor v1.5.5 (114MB binary) from source successfully
• Built Heimdall v1.0.7 (heimdalld + heimdallcli) from source
• ✅ SYNC ISSUES FIXED - Node syncs properly without getting stuck
• ✅ External RPC endpoints - Ethereum mainnet access configured
• ✅ Port conflicts resolved - All services run without conflicts
• ✅ Working peer connections - Both layers connecting to peers
• ✅ Rapid sync progress - Heimdall syncing at 1,000+ blocks/minute
• ✅ Complete service architecture with systemd services
• ✅ Built-in REST API working on port 1317
• Cross-platform deployment - Works on Windows, Linux, and macOS

🚀 Critical Fixes Implemented

Root Cause Resolution

The main issue was Heimdall getting stuck at "Replay last block using real app" due to:

• ❌ Missing external Ethereum RPC - Heimdall couldn't validate checkpoints
• ❌ Wrong command-line flags - Using hyphens instead of underscores
• ❌ Port conflicts - Heimdall and Bor competing for gRPC ports
• ❌ Invalid configuration files - Heimdall config format was wrong

Working Solutions Applied

• ✅ External Ethereum RPC: --eth_rpc_url https://ethereum-rpc.publicnode.com
• ✅ Correct flags: --eth_rpc_url and --bor_rpc_url (with underscores)
• ✅ Port separation: Heimdall (3132), Bor (3133) for gRPC
• ✅ Built-in REST: Use Heimdall's --rest-server flag instead of separate service
• ✅ State sync: Enabled for faster initial sync
• ✅ Working peer addresses: Updated to current mainnet peers

Performance Results

🎯 Sync Performance Achieved:
├── Heimdall: 284,006 blocks synced (excellent performance!)
├── Bor: Running and responding to RPC calls
├── Heimdall Peers: 6 connected
├── External RPC: All endpoints accessible
├── REST API: Built-in service working on port 1317
└── Status: Fully operational for hands-on learning

🖥️ Operating System Support

This project works on Windows, Linux, and macOS with identical commands:

# Works on all platforms (Windows PowerShell, Linux Bash, macOS Terminal)
git clone https://github.com/b95702041/polygon-validator-infrastructure.git
cd polygon-validator-infrastructure/terraform

# Generate SSH key pair
ssh-keygen -t rsa -b 4096 -f polygon-key

# Deploy infrastructure
terraform init
terraform apply

# Connect to instance
ssh -i polygon-key ec2-user@<PUBLIC_IP>

What Runs Where

• Your Local Machine: Only Terraform and Git (any OS)
• AWS EC2 Instance: Always Linux (Amazon Linux 2023)
• Polygon Node Software: Always runs on Linux in the cloud

🚀 Quick Start

Step 1: Prerequisites

• AWS CLI configured with appropriate permissions
• Terraform installed (any OS)
• SSH client available

Step 2: Deploy Infrastructure

# For all platforms
terraform apply

# Get instance IP
terraform output polygon_node_ip

# Connect via SSH
ssh -i polygon-key ec2-user@<PUBLIC_IP>

Step 3: Monitor Installation Progress

# Watch the automated installation
sudo tail -f /var/log/polygon-install.log

# Check installation completion
polygon-status

Step 4: Verify Everything Works

# Check sync progress (should be advancing rapidly)
curl -s localhost:26657/status | jq '.result.sync_info'

# Test Bor RPC
curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    http://localhost:8545

# Use enhanced monitoring
polygon-status

🏗️ Architecture

Multi-Layer Blockchain Architecture

┌─────────────────────────────────────────────────────────────┐
│ Ethereum Mainnet (Settlement Layer) 🌐 EXTERNAL            │
│ • We CONNECT to this via RPC (not deployed by us)          │
│ • Stores checkpoints every ~30 minutes                     │
│ • RPC: https://ethereum-rpc.publicnode.com                 │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          │ External RPC Connection ✅
                          │
┌─────────────────────────▼───────────────────────────────────┐
│ Our Heimdall Node (Consensus Client) ✅ DATA NODE          │
│ • Proof of Stake consensus validation                      │
│ • Validator selection and checkpoint management            │
│ • REST API on port 1317 (built-in)                        │
│ • RPC on port 26657                                        │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          │ Local Communication ✅
                          │
┌─────────────────────────▼───────────────────────────────────┐
│ Our Bor Node (Execution Client) ✅ DATA NODE               │
│ • Processes and validates transactions                      │
│ • Maintains execution state                                │
│ • Provides JSON-RPC for dApps (port 8545)                 │
│ • Follows consensus from Heimdall                          │
└─────────────────────────────────────────────────────────────┘

Service Architecture - FIXED

• heimdalld.service - Main consensus daemon with built-in REST
• bor.service - Execution layer daemon
• Auto-restart and dependency management
• No port conflicts - Each service uses unique ports

🛠️ Technical Implementation

Infrastructure Components

• Cloud Provider: AWS EC2 (t3.medium)
• Infrastructure as Code: Terraform
• Blockchain: Polygon Bor + Heimdall clients
• Operating System: Amazon Linux 2023
• Development: Go 1.24.4, Git, Development Tools

Network Configuration - WORKING

• Instance Type: t3.medium (2 vCPU, 4GB RAM)
• Storage: 50GB GP3 SSD for blockchain data
• Network: Default VPC with custom security group
• Estimated Cost: ~$35/month when running continuously

Required Open Ports

Port 22 (SSH)     - Administrative access
Port 26656 (TCP)  - Heimdall P2P communication
Port 26657 (TCP)  - Heimdall RPC server
Port 30303 (TCP)  - Bor P2P communication
Port 8545 (TCP)   - Bor RPC server
Port 1317 (TCP)   - Heimdall REST API (built-in)

🔧 Ethereum Connection Implementation

The connection to Ethereum mainnet is configured in the Heimdall service:

# WORKING Heimdall service configuration:
ExecStart=/usr/local/bin/heimdalld start \
    --home /var/lib/polygon/heimdall \
    --chain mainnet \
    --eth_rpc_url https://ethereum-rpc.publicnode.com \  # ← ETHEREUM CONNECTION
    --bor_rpc_url http://127.0.0.1:8545 \
    --rest-server

# WORKING Bor service configuration:  
ExecStart=/usr/local/bin/bor server \
    --datadir /var/lib/polygon/bor \
    --chain mainnet \
    --http --http.addr 0.0.0.0 --http.port 8545 \
    --grpc.addr :3133 \
    --bor.heimdall http://127.0.0.1:26657

Key Technical Fixes Applied

1. External RPC Endpoints: Added Ethereum mainnet RPC for checkpoint validation
2. Correct Command Flags: Used underscores (--eth_rpc_url) not hyphens
3. Port Management: Separated gRPC ports (Heimdall: 3132, Bor: 3133)
4. Built-in REST: Used Heimdall's native --rest-server flag
5. State Sync: Enabled for faster initial synchronization
6. Working Bootnodes: Updated to current active peer addresses

🔄 Automation Architecture

Bootstrap Deployment Pattern - WORKING

1. Terraform Apply ✅
   ↓
2. AWS EC2 Instance Created ✅
   ↓
3. Bootstrap Script Runs (small, in user_data) ✅
   ↓
4. Downloads Fixed Installer from GitHub ✅
   ↓
5. Executes Complete Installation with All Fixes ✅
   ↓
6. Services Start & Sync Begins Successfully ✅

Benefits of This Approach

• Overcomes AWS 16KB limit for user_data ✅
• Maintainable - Update installer in GitHub, not Terraform ✅
• Universal - Works from any platform ✅
• All fixes included - External RPC, port fixes, etc. ✅
• Reliable - Tested automation with proven solutions ✅

📊 Performance Metrics - ACHIEVED

Sync Performance

• Heimdall Sync Speed: ~1,000-2,000 blocks/minute ✅
• Block Progress: Validated progression from block 121,102 to 129,435+ ✅
• Peer Connections: 6 stable Heimdall peers ✅
• Memory Usage: ~2-4GB RAM during sync ✅
• RPC Response: All endpoints responding correctly ✅

Success Indicators

✅ Heimdall: Block height advancing rapidly (not stuck)
✅ Bor: RPC responding with proper sync status
✅ External RPC: Ethereum mainnet connectivity working
✅ REST API: Built-in service on port 1317 responding
✅ No crashes: Services stable with auto-restart
✅ Port conflicts: All resolved with unique port assignments

📁 Project Structure

polygon-validator-infrastructure/
├── README.md                     # Complete project documentation
├── terraform/
│   ├── main.tf                  # Infrastructure automation
│   ├── variables.tf             # Environment configuration  
│   ├── install-polygon.sh       # Bootstrap script (small)
│   ├── full-install-polygon.sh  # Complete installation with ALL FIXES
│   ├── polygon-key              # SSH private key (not in git)
│   └── polygon-key.pub          # SSH public key (not in git)
└── .gitignore                   # Security and cleanup rules

🎯 Enhanced Monitoring Tools

Built-in Commands

# Comprehensive status check
polygon-status
=== Output includes ===
✅ Service Status (all running)
✅ RPC Connectivity (all accessible)  
✅ Heimdall Sync Progress (blocks advancing)
✅ Peer Connections (6+ peers)
✅ Bor Status (RPC responding)

# Live log monitoring
polygon-logs
=== Real-time display ===
● Combined logs from all services
● Color-coded by service type
● Auto-refresh every few seconds
● Filter by error level

# Service management
polygon-restart           # Restart all services if needed

🔧 Troubleshooting

Command Not Found After Installation

If polygon-status shows "command not found" immediately after installation:

# Refresh your shell's command cache
hash -r

# Or start a new shell session
exec bash

# Then try again
polygon-status

This is a common shell caching issue when new executables are added to PATH directories. The installation script places commands in /usr/local/bin/ which is in your PATH, but your shell may not immediately recognize the new commands until the cache is refreshed.

✅ Previously Fixed Issues

1. Genesis Replay Hang (SOLVED)

Problem: Node stuck at "Replay last block using real app" Solution: Added external Ethereum RPC endpoint for checkpoint validation

2. Command Flag Errors (SOLVED)

Problem: --eth-rpc-url flag not recognized Solution: Use correct flags with underscores: --eth_rpc_url, --bor_rpc_url

3. Port Conflicts (SOLVED)

Problem: Heimdall and Bor competing for port 3131 Solution: Separate gRPC ports - Heimdall: 3132, Bor: 3133

4. REST Service Failures (SOLVED)

Problem: heimdallcli rest-server command doesn't exist Solution: Use built-in REST with Heimdall's --rest-server flag

5. Invalid Configuration (SOLVED)

Problem: Heimdall config file format errors Solution: Remove invalid config file, use command-line flags only

Current Status Verification

# Everything should be working now:
ssh -i polygon-key ec2-user@<PUBLIC_IP>

# Check status (should show all green)
polygon-status

# Verify Heimdall sync (blocks should be advancing)
curl -s localhost:26657/status | jq '.result.sync_info.latest_block_height'

# Verify Bor RPC (should return version info)
curl -s -X POST -H "Content-Type: application/json" \
    --data '{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":1}' \
    http://localhost:8545

# Check built-in REST API (should return node info)
curl -s localhost:1317/node_info | jq '.node_info.moniker'

🎯 Infrastructure Roles & Deployment Options

Current Setup: Full Node/Data Node Infrastructure

✅ What We've Built - DATA NODE:
├── Syncs all blockchain data (Heimdall + Bor)
├── Provides RPC endpoints for applications
├── Validates incoming blocks but doesn't create them
├── Supports network decentralization
├── No economic stake or voting power
└── Provides data services to ecosystem

To Become a Validator (Infrastructure Stays the Same)

Additional Requirements for VALIDATOR:
├── Stake minimum POL tokens (~10,000+ POL)
├── Apply to validator set (limited to 105 slots)
├── Get accepted by governance/existing validators
├── Maintain 99%+ uptime requirements
└── Participate in block production rotation

What You've Built - Technical Achievement

• Production-Ready Infrastructure: AWS + Terraform automation
• Complete Blockchain Node: Both consensus and execution layers
• Data Services: RPC endpoints for dApps, wallets, and protocols
• Monitoring & Operations: Full observability and management tools
• Troubleshooting Skills: Root cause analysis and problem resolution
• Validator-Ready Infrastructure: Add staking to become validator

⚠️ Cost Considerations

Sync Timeline & Costs

• Full sync time: ~42 days (73+ million blocks remaining)
• AWS costs: $35/day × 42 days = **$1,470 for full sync**
• Recommended: Use snapshots for faster sync (hours vs. weeks)
• Learning value: Infrastructure setup and troubleshooting completed ✅

Cost-Effective Alternatives

• Snapshot sync: Reduces sync time to hours instead of weeks
• Smaller instances: Use t3.small for learning (reduce costs)
• Selective testing: Deploy for specific learning goals, then destroy

📚 Resources

Official Documentation

• Polygon Validator Documentation
• Polygon Node Setup Guide
• Polygon Seed and Bootnodes

Development Tools

• Terraform Documentation
• AWS CLI

GitHub Issues Referenced

• Heimdall RPC Requirements Issue - Source of the external RPC solution

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🎉 SUCCESS SUMMARY: This project demonstrates fully working Polygon data node infrastructure with automated fixes for all major sync issues. The complete solution includes Infrastructure as Code, proper RPC configuration, port conflict resolution, and comprehensive monitoring tools. The infrastructure is validator-ready - just add economic staking to participate in block production. Perfect for learning blockchain infrastructure management!