Read RPC

This workspace holds the collection of components for the Read RPC.

Branch Information

• main - Latest released version, stable and production-ready
• develop - Development branch containing the latest changes and features under development

NEAR Dependencies

This project relies heavily on NEAR ecosystem crates and uses dependencies directly from Git repositories rather than published crates from crates.io. This approach is necessary because:

1. Unpublished Crates: NEAR doesn't publish all the necessary crates required for Read RPC functionality to public registries
2. Version Compatibility: We need specific versions or features that may not be available in published releases
3. Custom Modifications: Some dependencies require modifications to work optimally with Read RPC

Forked Dependencies

• near-jsonrpc-client - Provides JSON RPC client functionality for communicating with NEAR nodes
• near-lake-framework - Essential for indexing blockchain data from NEAR Lake (AWS S3 data buckets)

Fork Maintenance

These dependencies are maintained as forks to:

• Ensure compatibility with Read RPC's specific requirements
• Include necessary modifications and patches
• Control the update cycle and maintain stability
• Add features or fixes that haven't been merged upstream yet

Dependency Management

When updating NEAR dependencies:

1. Check for upstream changes in the original repositories
2. Test compatibility with existing Read RPC functionality
3. Update fork references in Cargo.toml files
4. Ensure all components still build and function correctly

Current content

rpc-server

The JSON RPC server implementation that repeats all the APIs current real NEAR JSON RPC but using a different data sources:

• The Read RPC Storage (PostgresDB currently)
• NEAR Lake Data buckets (AWS S3 currently)
• real NEAR JSON RPC

state-indexer

The indexer built on top of Lake Framework that watches the network and stores the StateChanges into the Storage using the designed data schemas.

near-state-indexer

The indexer built on NEAR Indexer Framework

tx-indexer

The indexer built on top of Lake Framework that watches the network and stores the Transactions along with all the related entities (Receipts, ExecutionOutcomes) into the Storage using the specifically defined TransactionDetails structure in a dumped way (using the simplest key-value schema)

config

The configuration module is responsible for managing the configuration settings of the NEAR ReadRPC project.

database

The database module provides database abstractions and implementations for storing and retrieving data.

cache-storage

The cache storage module provides caching functionality for improved performance.

tx-details-storage

The transaction details storage module handles storage of detailed transaction information.

logic-state-indexer

The logic state indexer module provides state indexing functionality.

readnode-primitives

The readnode primitives module contains common data structures and utilities.

perf-testing

The performance testing module provides tools for testing and benchmarking.

Documentation

Project Documentation

• CHANGELOG.md - Project changelog and version history
• Examples - Usage examples and sample configurations

Technical Documentation

• RPC Methods - Available RPC methods and their specifications
• Custom RPC Methods - Custom RPC methods specific to Read RPC
• Database Migrations - Database migration procedures and guidelines
• Tracing - Distributed tracing setup and configuration

Database Documentation

• PostgreSQL Setup - PostgreSQL-specific configuration and setup
• Database Migrations - Database migration procedures and guidelines

Docker compose

Note! The docker compose is not fully ready yet. It's still in progress. However, you can run the entire project to play around with it. It is still not convenient for development or debugging purposes. We are working on improving it.

Run the entire project

Put TOML file config.toml with configuration in the home root of the project. See the example here.

Run the docker compose:

$ docker-compose up

This will spin up:

• read-rpc-server - the JSON RPC server
• state-indexer - the indexer that watches the network and stores the StateChanges into the Storage using the designed data schemas.
• tx-indexer - the indexer that watches the network and stores the Transactions along with all the related entities (Receipts, ExecutionOutcomes) into the Storage using the specifically defined TransactionDetails structure in a dumped way (using the simplest key-value schema)
• jaeger - the Jaeger instance for tracing (http://localhost:16686)

Examples:

See examples here.

TODO:

• Setup all the services to use volumes for the codebase and using cargo watch to recompile the code on the fly
• Come up with the way of changin the configuration for the indexers in a convenient way
• Robust documentation for the docker compose setup

Development

Check the entire product

cargo check

Fmt the entire product

cargo fmt --all

Run a specific package

cargo run --package <package_name>

Example:

cargo run --profile dev-release --package rpc-server

Please don't forget to handle .env files properly if you're running the package from the root using the proposed approach.

Keep in mind

You can move to the folder with a package

cd rpc-server/

And run a package as usually from the folder

cargo run --profile dev-release

In this case you would want to have an .env file in the package folder.

Tracing

See the tracing documentation here