Skip to content

Improve data client for blockchain adapter #2787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 22 commits into from
Jul 14, 2025
Merged

Improve data client for blockchain adapter #2787

merged 22 commits into from
Jul 14, 2025

Conversation

filipmacek
Copy link
Member

@filipmacek filipmacek commented Jul 14, 2025 •
edited
Loading

Pull Request

NautilusTrader prioritizes correctness and reliability, please follow existing patterns for validation and testing.

Summary

  • Added PostgreSQL cache database support directly to BlockchainDataClientConfig, eliminating manual initialization
  • Implemented batch block inserts using PostgreSQL UNNEST reducing database round trips and increased max database connections
  • Standardized DEX naming conventions, removing spaces and using camelCase (UniswapV3 instead of Uniswap V3)
  • Centralized DEX lookup logic replacing hardcoded string keys with dynamic IDs
  • Added factory deployment block numbers to all DEX implementations
  • Added dex_ids parameter to blockchain config for DEX registration
  • Implemented comprehensive sync performance tracking with progress, ETA, and timing metrics
  • Added configurable batch processing with 1000 blocks per batch for optimized memory usage
  • Enhanced ERC20 token processing to handle missing metadata gracefully
  • Added Unicode symbol sanitization, normalizing currency symbols to ASCII equivalents
  • Improved HTTP RPC error handling for better resilience against failed requests
  • Implemented graceful degradation skipping problematic tokens instead of failing entire operations (tokens that don't conform to the ERC20 standard or are deployed on a different chain)
  • Remove Hypersync subscription logic , simplifying client architecture, and added TODO to implement it in the data client
  • Blockchain(EVM) adapter #1335

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (impacts existing behavior)
  • Documentation update
  • Maintenance / chore

filipmacek added 21 commits July 14, 2025 16:47
@filipmacek
Add Postgres cache database support to blockchain data clients config
420da56
@filipmacek
Refactor DEX exchange structure and update naming conventions
35a7406
@filipmacek
Add a factory deployment block number to Dex struct
55a8e11
@filipmacek
Remove Hypersync client indexing and subscriptions
136904c
@filipmacek
Fix factory creation block
0fec222
@filipmacek
Fix dex exchange structure
49e7a34
@filipmacek
Fix postgres cache database support
96b2032
@filipmacek
Add dex_ids param to BlockchainDataClientConfig
648c896
@filipmacek
Investigate initial performance tracking for syncing process
74943f3
@filipmacek
Move reporting into a separate function
ba3d7ea
@filipmacek
Add blocks batch inserts
da044bb
@filipmacek
Add to_block option in sync_blocks
bc7272a
@filipmacek
Remove full chain sync time estimation from log output
87c423d
@filipmacek
Improve connect function blockchain data client
d4aa61e
@filipmacek
Reuse pool parsing for UniswapV3 in Arbitrum from Ethereum
857a575
@filipmacek
Improve HTTP RPC error handling
df5847f
@filipmacek
Sanitize token symbols in AMM to handle special characters
feb6d1e
@filipmacek
Improve token processing and handle missing ERC20 metadata
dec34b0
@filipmacek
Add TODO to implement subscription logic in blockchain data client
a2c5751
@filipmacek
Fix format lints
6d76c24
@filipmacek
Optimize load_block_timestamps query removing DISTINCT ON clause
4bd43ae
@filipmacek filipmacek requested a review from cjdsellers July 14, 2025 15:05
@filipmacek filipmacek self-assigned this Jul 14, 2025
@filipmacek filipmacek added the rust Relating to the Rust core label Jul 14, 2025
cjdsellers
cjdsellers approved these changes Jul 14, 2025
View reviewed changes
Copy link
Member

@cjdsellers cjdsellers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@cjdsellers cjdsellers merged commit 97980ed into develop Jul 14, 2025
13 checks passed
@cjdsellers cjdsellers deleted the blockchain-adapter-sync branch July 14, 2025 23:01
Bshara23 pushed a commit to Bshara23/nautilus_trader that referenced this pull request Sep 5, 2025
@filipmacek
Improve data client for blockchain adapter (nautechsystems#2787)
2906888 
- Add Postgres cache database support to blockchain data clients config
- Refactor DEX exchange structure and update naming conventions
- Add a factory deployment block number to `Dex` struct
- Fix factory creation block
- Fix DEX exchange structure
- Fix Postgres cache database support
- Add `dex_ids` param to `BlockchainDataClientConfig`
- Investigate initial performance tracking for syncing process
- Move reporting into a separate function
- Add blocks batch inserts
- Add `to_block` option in `sync_blocks`
- Remove full chain sync time estimation from log output
- Improve connect function blockchain data client
- Reuse pool parsing for UniswapV3 in Arbitrum from Ethereum
- Improve HTTP RPC error handling
- Sanitize token symbols in AMM to handle special characters
- Improve token processing and handle missing ERC20 metadata
- Optimize `load_block_timestamps` query removing DISTINCT ON clause
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cjdsellers cjdsellers cjdsellers approved these changes

Assignees

@filipmacek filipmacek

Labels
rust Relating to the Rust core
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@filipmacek @cjdsellers