Modular block request handler by rahulksnv · Pull Request #1524 · paritytech/polkadot-sdk

rahulksnv · 2023-09-12T18:57:44Z

Submit the outstanding PRs from the old repos(these were already reviewed and approved before the repo rorg, but not yet submitted):
Main PR: paritytech/substrate#14014
Companion PRs: paritytech/polkadot#7134, paritytech/cumulus#2489

The changes in the PR:

ChainSync currently calls into the block request handler directly. Instead, move the block request handler behind a trait. This allows new protocols to be plugged into ChainSync.
BuildNetworkParams is changed so that custom relay protocol implementations can be (optionally) passed in during network creation time. If custom protocol is not specified, it defaults to the existing block handler
BlockServer and BlockDownloader traits are introduced for the protocol implementation. The existing block handler has been changed to implement these traits
Other changes:
[X] Make TxHash serializable. This is needed for exchanging the serialized hash in the relay protocol messages
[X] Clean up types no longer used(OpaqueBlockRequest, OpaqueBlockResponse)

altonen · 2023-09-13T09:54:35Z

I haven't looked at the changes yet but this may have to wait a little while since @dmitry-markin is doing refactoring on ChainSync, especially w.r.t. to request handling.

nazar-pc · 2023-09-13T14:07:20Z

PR was out at the old repo since April, I was really hoping we can upstream this since it touches pretty fundamental internals and every time something changes upstream it is a bit of PITA to rebase it.

altonen · 2023-09-13T16:03:15Z

Understandable and apologies for stalling with the review of the original PR. We've finally found time to start refactoring the syncing code which is long overdue and the idea is to convert ChainSync to do no I/O or polling, which, after a brief look, is at odds with the changes of this PR. We're most likely doing a facelift for the request/response code relatively soon as well.

@dmitry-markin can you give an estimate how much work is it to rebase your changes on top of these changes?

dmitry-markin · 2023-09-14T09:21:56Z

@dmitry-markin can you give an estimate how much work is it to rebase your changes on top of these changes?

I think we can merge this, the changes of this PR are mostly self contained and don't interfere in a bad way (and even simplify some things) with work on extracting request-responses. I'll just move the BlockDownloader one level up to the SyncingEngine.

As a side note, the situation around this PR seems quite weird, because the original PR was approved a long time ago, just nobody pressed the "merge" button.

rahulksnv · 2023-09-14T15:51:32Z

As a side note, the situation around this PR seems quite weird, because the original PR was approved a long time ago, just nobody pressed the "merge" button.

Apologies on that. I had a hard time getting green on all three PRs (the main one + two in the companion repos) .. by the time I get green, something would have changed and had to resync/retry. At some point I got busy with other stuff and this went to the back burner. So glad that the companion repos have been consolidated into one repo now

Patch the outstanding PRs from the old repos: paritytech/substrate#14014 paritytech/polkadot#7134 paritytech/cumulus#2489 These were already reviewed and approved, but not yet submitted.

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

rahulksnv · 2023-09-15T04:47:50Z

@dmitry-markin @altonen Addressed the comments, please let me know what else is needed in order to merge the PR, thanks.

dmitry-markin · 2023-09-15T06:49:31Z

bot fmt

command-bot · 2023-09-15T06:49:36Z

@dmitry-markin https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3704149 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh". Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 5-6f203b54-1fd3-419a-b655-322cd76803f4 to cancel this command or bot cancel to cancel all commands in this pull request.

command-bot · 2023-09-15T06:53:02Z

@dmitry-markin Command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh" has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3704149 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3704149/artifacts/download.

nazar-pc · 2023-09-15T08:13:52Z

Thanks you for quick progress here!

dmitry-markin · 2023-09-15T08:33:23Z

@dmitry-markin @altonen Addressed the comments, please let me know what else is needed in order to merge the PR, thanks.

All is good, the PR is merged. Thanks for contributing!

rahulksnv · 2023-09-15T16:51:51Z

@dmitry-markin @altonen Addressed the comments, please let me know what else is needed in order to merge the PR, thanks.

All is good, the PR is merged. Thanks for contributing!

Thanks @dmitry-markin @altonen for the help in getting this merged

* update dependencies * reflect paritytech/polkadot-sdk#1524

Polkadot-Forum · 2023-10-27T17:04:32Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-release-analysis-v1-2-0/4451/1

Submit the outstanding PRs from the old repos(these were already reviewed and approved before the repo rorg, but not yet submitted): Main PR: paritytech/substrate#14014 Companion PRs: paritytech/polkadot#7134, paritytech/cumulus#2489 The changes in the PR: 1. ChainSync currently calls into the block request handler directly. Instead, move the block request handler behind a trait. This allows new protocols to be plugged into ChainSync. 2. BuildNetworkParams is changed so that custom relay protocol implementations can be (optionally) passed in during network creation time. If custom protocol is not specified, it defaults to the existing block handler 3. BlockServer and BlockDownloader traits are introduced for the protocol implementation. The existing block handler has been changed to implement these traits 4. Other changes: [X] Make TxHash serializable. This is needed for exchanging the serialized hash in the relay protocol messages [X] Clean up types no longer used(OpaqueBlockRequest, OpaqueBlockResponse) --------- Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: command-bot <>

* Setup deps * Remove Koi from account migration test * paritytech/polkadot-sdk#1495 * Bump * paritytech/polkadot-sdk#1524 * !! paritytech/polkadot-sdk#1363 * paritytech/polkadot-sdk#1492 * paritytech/polkadot-sdk#1911 * paritytech/polkadot-sdk#1900 Signed-off-by: Xavier Lau <xavier@inv.cafe> * paritytech/polkadot-sdk#1661 * paritytech/polkadot-sdk#2144 * paritytech/polkadot-sdk#2048 * paritytech/polkadot-sdk#1672 * paritytech/polkadot-sdk#2303 * paritytech/polkadot-sdk#1256 * Remove identity and vesting * Fixes * paritytech/polkadot-sdk#2657 * paritytech/polkadot-sdk#1313 * paritytech/polkadot-sdk#2331 * paritytech/polkadot-sdk#2409 part.1 * paritytech/polkadot-sdk#2767 * paritytech/polkadot-sdk#2521 Signed-off-by: Xavier Lau <xavier@inv.cafe> * paritytech/polkadot-sdk#1222 * paritytech/polkadot-sdk#1234 part.1 * Satisfy compiler * XCM V4 part.1 * paritytech/polkadot-sdk#1246 * Remove pallet-democracy part.1 * paritytech/polkadot-sdk#2142 * paritytech/polkadot-sdk#2428 * paritytech/polkadot-sdk#3228 * XCM V4 part.2 * Bump * Build all runtimes * Build node * Remove pallet-democracy Signed-off-by: Xavier Lau <xavier@inv.cafe> * Format * Fix pallet tests * Fix precompile tests * Format * Fixes * Async, remove council, common pallet config * Fix `ethtx-forward` test case (#1519) * Fix ethtx-forward tests * Format * Fix following the review * Fixes * Fixes * Use default impl * Benchmark helper * Bench part.1 * Bench part.2 * Bench part.3 * Fix all tests * Typo * Feat * Fix EVM tracing build * Reuse upstream `proof_size_base_cost()` (#1521) * Format issue * Fixes * Fix CI --------- Signed-off-by: Xavier Lau <xavier@inv.cafe> Co-authored-by: Bear Wang <boundless.forest@outlook.com>

* update dependencies * reflect paritytech/polkadot-sdk#1524

* Setup deps * Remove Koi from account migration test * paritytech/polkadot-sdk#1495 * Bump * paritytech/polkadot-sdk#1524 * !! paritytech/polkadot-sdk#1363 * paritytech/polkadot-sdk#1492 * paritytech/polkadot-sdk#1911 * paritytech/polkadot-sdk#1900 Signed-off-by: Xavier Lau <xavier@inv.cafe> * paritytech/polkadot-sdk#1661 * paritytech/polkadot-sdk#2144 * paritytech/polkadot-sdk#2048 * paritytech/polkadot-sdk#1672 * paritytech/polkadot-sdk#2303 * paritytech/polkadot-sdk#1256 * Remove identity and vesting * Fixes * paritytech/polkadot-sdk#2657 * paritytech/polkadot-sdk#1313 * paritytech/polkadot-sdk#2331 * paritytech/polkadot-sdk#2409 part.1 * paritytech/polkadot-sdk#2767 * paritytech/polkadot-sdk#2521 Signed-off-by: Xavier Lau <xavier@inv.cafe> * paritytech/polkadot-sdk#1222 * paritytech/polkadot-sdk#1234 part.1 * Satisfy compiler * XCM V4 part.1 * paritytech/polkadot-sdk#1246 * Remove pallet-democracy part.1 * paritytech/polkadot-sdk#2142 * paritytech/polkadot-sdk#2428 * paritytech/polkadot-sdk#3228 * XCM V4 part.2 * Bump * Build all runtimes * Build node * Remove pallet-democracy Signed-off-by: Xavier Lau <xavier@inv.cafe> * Format * Fix pallet tests * Fix precompile tests * Format * Fixes * Async, remove council, common pallet config * Fix `ethtx-forward` test case (#1519) * Fix ethtx-forward tests * Format * Fix following the review * Fixes * Fixes * Use default impl * Benchmark helper * Bench part.1 * Bench part.2 * Bench part.3 * Fix all tests * Typo * Feat * Fix EVM tracing build * Reuse upstream `proof_size_base_cost()` (#1521) * Format issue * Fixes * Fix CI --------- Signed-off-by: Xavier Lau <xavier@inv.cafe> Co-authored-by: Bear Wang <boundless.forest@outlook.com>

rahulksnv force-pushed the modular-block-handler branch from 3e6dd57 to 7096f91 Compare September 12, 2023 19:49

rahulksnv marked this pull request as ready for review September 12, 2023 19:50

bkchr requested a review from a team September 13, 2023 07:42

dmitry-markin approved these changes Sep 14, 2023

View reviewed changes

Comment thread substrate/client/network/sync/src/lib.rs Outdated

Comment thread substrate/client/network/sync/src/lib.rs Outdated

Comment thread substrate/client/network/sync/src/block_relay_protocol.rs Outdated

dmitry-markin requested a review from altonen September 14, 2023 09:22

dmitry-markin added the T0-node This PR/Issue is related to the topic “node”. label Sep 14, 2023

altonen approved these changes Sep 14, 2023

View reviewed changes

Comment thread substrate/client/network/sync/src/lib.rs

rahulksnv and others added 5 commits September 14, 2023 10:29

Modular block request handler

8b6bb6b

Patch the outstanding PRs from the old repos: paritytech/substrate#14014 paritytech/polkadot#7134 paritytech/cumulus#2489 These were already reviewed and approved, but not yet submitted.

Update substrate/client/network/sync/src/lib.rs

8ed63ba

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Update substrate/client/network/sync/src/lib.rs

776faea

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Update substrate/client/network/sync/src/block_relay_protocol.rs

dbbeee7

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Update method name

2fcbba7

rahulksnv force-pushed the modular-block-handler branch from 3aeb10a to 2fcbba7 Compare September 14, 2023 17:35

Merge branch 'master' into modular-block-handler

9f7fac6

".git/.scripts/commands/fmt/fmt.sh"

7508e8f

Merge branch 'master' into modular-block-handler

9b9a69d

dmitry-markin enabled auto-merge (squash) September 15, 2023 07:00

dmitry-markin merged commit b35b28c into paritytech:master Sep 15, 2023

kiltbot mentioned this pull request Oct 19, 2023

[AUTOMATIC] Update Polkadot dependencies from 1.1.0 to 1.2.0 KILTprotocol/kilt-node#571

Closed

JoshOrndorff pushed a commit to Off-Narrative-Labs/Tuxedo that referenced this pull request Oct 23, 2023

reflect paritytech/polkadot-sdk#1524

33a9d86

JoshOrndorff added a commit to Off-Narrative-Labs/Tuxedo that referenced this pull request Oct 23, 2023

Update substrate dependency to polkadot-v1.2.0 (#134)

2770b34

* update dependencies * reflect paritytech/polkadot-sdk#1524

ahmadkaouk mentioned this pull request Nov 16, 2023

Update polkadot-sdk from v.1.1.0 to v1.3.0 moonbeam-foundation/moonbeam#2565

Closed

dmitry-markin mentioned this pull request Dec 15, 2023

Extract warp sync strategy from ChainSync #2467

Merged

dnjscksdn98 mentioned this pull request Jan 12, 2024

NODE-65, deps: polkadot sdk migration bifrost-platform/bifrost-node#36

Merged

13 tasks

Dinonard mentioned this pull request Mar 28, 2024

Feat/polkadot v1.3.0 uplift AstarNetwork/Astar#1207

Merged

aurexav added a commit to darwinia-network/darwinia that referenced this pull request Jun 21, 2024

paritytech/polkadot-sdk#1524

b86f455

nazar-pc mentioned this pull request Nov 11, 2024

Fix wasteful bandwidth usage by transaction gossip #6433

Open

2 tasks

aiden94a added a commit to aiden94a/Tuxedo that referenced this pull request Sep 7, 2025

Update substrate dependency to polkadot-v1.2.0 (#134)

91d1497

* update dependencies * reflect paritytech/polkadot-sdk#1524

Conversation

rahulksnv commented Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

altonen commented Sep 13, 2023

Uh oh!

nazar-pc commented Sep 13, 2023

Uh oh!

altonen commented Sep 13, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmitry-markin commented Sep 14, 2023

Uh oh!

Uh oh!

rahulksnv commented Sep 14, 2023

Uh oh!

rahulksnv commented Sep 15, 2023

Uh oh!

dmitry-markin commented Sep 15, 2023

Uh oh!

command-bot Bot commented Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

command-bot Bot commented Sep 15, 2023

Uh oh!

nazar-pc commented Sep 15, 2023

Uh oh!

dmitry-markin commented Sep 15, 2023

Uh oh!

rahulksnv commented Sep 15, 2023

Uh oh!

Polkadot-Forum commented Oct 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rahulksnv commented Sep 12, 2023 •

edited

Loading

command-bot Bot commented Sep 15, 2023 •

edited

Loading