collator-side: send parent head data by ordian · Pull Request #3521 · paritytech/polkadot-sdk

ordian · 2024-02-29T11:05:08Z

On top of #3302.

We want the validators to upgrade first before we add changes to the collation side to send the new variants, which is why this part is extracted into a separate PR.

The detection of when to send the parent head is based on the core assignments at the relay parent of the candidate. We probably want to make it more flexible in the future, but for now, it will work for a simple use case when a para always has multiple cores assigned to it.

This feature is currently gated by elastic-scaling-experimental feature flag.

* master: (41 commits) Add Coretime to Westend (#3319) removed `pallet::getter` from `pallet-sudo` (#3370) gossip-support: add unittests for update authorities (#3258) [FRAME Core] remove unnecessary overrides while using derive_impl for frame_system (#3317) Update coretime-westend bootnodes (#3380) `im-online` removal cleanup: remove off-chain storage (#2290) Bump the known_good_semver group with 1 update (#3379) Fix documentation dead link (#3372) Make `sp-keystore` `no_std`-compatible and fix the `build-runtimes-polkavm` CI job (#3363) Remove unused `im-online` weights (#3373) Ensure referenda `TracksInfo` is sorted (#3325) rpc server: add rate limiting middleware (#3301) do not block finality for "disabled" disputes (#3358) fix(zombienet): docker `img` version to use in merge queues for bridges (#3337) Various nits and alignments for SP testnets found during bumping `polkadot-fellows` repo (#3359) Add broker pallet to `coretime-westend` (#3272) remove recursion limit (#3348) Update subkey README.md (#3355) Bump the known_good_semver group with 6 updates (#3347) Document LocalKeystore insert method (#3336) ...

…head-data * origin/master: (51 commits) Runtime Upgrade ref docs and Single Block Migration example pallet (#1554) Collator overseer builder unification (#3335) Introduce storage attr macro `#[disable_try_decode_storage]` and set it on `System::Events` and `ParachainSystem::HostConfiguration` (#3454) Add Polkadotters bootnoders per IBP application (#3423) Add documentation around FRAME Origin (#3362) Bridge zombienet tests: Check amount received at destination (#3490) Snowbridge benchmark tests fix (#3424) fix(zombienet): increase timeout in download artifacts (#3376) Cleanup String::from_utf8 (#3446) [prdoc] Validate crate names (#3467) Limit max execution time for `test-linux-stable` CI jobs (#3483) Introduce Notification block pinning limit (#2935) frame-support: Improve error reporting when having too many pallets (#3478) add Encointer as trusted teleporter for Westend (#3411) [pallet-xcm] Adjust benchmarks (teleport_assets/reserve_transfer_assets) not relying on ED (#3464) Add more debug logs to understand if statement-distribution misbehaving (#3419) Remove redundant parachains assigner pallet. (#3457) Use generic hash for runtime wasm in resolve_state_version_from_wasm (#3447) Runtime: allow backing multiple candidates of same parachain on different cores (#3231) Bridge zombienet tests: move all "framework" files under one folder (#3462) ...

When elastic scaling is detected (at the relay parent state), meaning more than one is assigned to the para at the time, the collator will send the collation variant with parent head data.

…head-data * origin/master: Fix call enum's metadata regression (#3513) Enable elastic scaling node feature in local testnets genesis (#3509) update development setup in sdk-docs (#3506) Fix accidental no-shows on node restart (#3277) Remove `AssignmentProviderConfig` and use parameters from `HostConfiguration` instead (#3181) [Deprecation] Remove sp_weights::OldWeight (#3491) Fixup multi-collator parachain transition to async backing (#3510) Multi-Block-Migrations, `poll` hook and new System callbacks (#1781) Snowbridge - Extract Ethereum Chain ID (#3501) PVF: re-preparing artifact on failed runtime construction (#3187) Add documentation around FRAME Offchain workers (#3463) [prdoc] Optional SemVer bumps and Docs (#3441) rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193)

…data * ao-collator-parent-head-data: Fix call enum's metadata regression (#3513) Enable elastic scaling node feature in local testnets genesis (#3509) update development setup in sdk-docs (#3506) Fix accidental no-shows on node restart (#3277) Remove `AssignmentProviderConfig` and use parameters from `HostConfiguration` instead (#3181) [Deprecation] Remove sp_weights::OldWeight (#3491) Fixup multi-collator parachain transition to async backing (#3510) Multi-Block-Migrations, `poll` hook and new System callbacks (#1781) Snowbridge - Extract Ethereum Chain ID (#3501) PVF: re-preparing artifact on failed runtime construction (#3187) Add documentation around FRAME Offchain workers (#3463) [prdoc] Optional SemVer bumps and Docs (#3441) rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193)

sandreim

🚀

* master: Finish documenting `#[pallet::xxx]` macros (#2638) Remove `as frame_system::DefaultConfig` from the required syntax in `derive_impl` (#3505) provisioner: allow multiple cores assigned to the same para (#3233) subsystem-bench: add regression tests for availability read and write (#3311) make SelfParaId a metadata constant (#3517) Fix crash of synced parachain node run with `--sync=warp` (#3523) [Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508) Add `claim_assets` extrinsic to `pallet-xcm` (#3403)

…data * ao-collator-parent-head-data: add a comment (review) Finish documenting `#[pallet::xxx]` macros (#2638) Remove `as frame_system::DefaultConfig` from the required syntax in `derive_impl` (#3505) provisioner: allow multiple cores assigned to the same para (#3233) subsystem-bench: add regression tests for availability read and write (#3311) make SelfParaId a metadata constant (#3517) Fix crash of synced parachain node run with `--sync=warp` (#3523) [Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508) Add `claim_assets` extrinsic to `pallet-xcm` (#3403)

* master: (65 commits) collator protocol changes for elastic scaling (validator side) (#3302) Contracts use polkavm workspace deps (#3715) Add elastic scaling support in ParaInherent BenchBuilder (#3690) Removes `as [disambiguation_path]` from `derive_impl` usage (#3652) fix(paseo-spec): New Paseo Bootnodes (#3674) Improve Penpal runtime + emulated tests (#3543) Staking ledger bonding fixes (#3639) DescribeAllTerminal for HashedDescription (#3349) Increase timeout for assertions (#3680) Add subsystems regression tests to CI (#3527) Always print connectivity report (#3677) Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665) authority-discovery: Add log for debugging DHT authority records (#3668) Construct Runtime v2 (#1378) Support for `keyring` in runtimes (#2044) Add api-name in `cannot query the runtime API version` warning (#3653) Add a PolkaVM-based executor (#3458) Adds default config for assets pallet (#3637) Bump handlebars from 4.3.7 to 5.1.0 (#3248) [Collator Selection] Fix weight refund for `set_candidacy_bond` (#3643) ...

alindima · 2024-03-18T12:43:51Z

I guess this should be marked as draft to prevent it from being merged and released until validators are upgraded to understand this protocol message?
From my understanding, the only way of currently preventing this is by not ordering multiple cores for the same paraid (but this is already happening in the wild). Alternatively, we could add a knob for enabling this, like a command-line flag

sandreim

@ordian I think we can safely merge this and not block testing, please see my comment.

@ggwpez

Part of #3326 @ggwpez @kianenigma @shawntabrizi polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp --------- Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>

Bumps the known_good_semver group with 2 updates: [serde_yaml](https://github.com/dtolnay/serde-yaml) and [syn](https://github.com/dtolnay/syn). Updates `serde_yaml` from 0.9.32 to 0.9.33 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/serde-yaml/releases">serde_yaml's releases</a>.</em></p> <blockquote> <h2>0.9.33</h2> <ul> <li>Fix quadratic parse time for YAML containing deeply nested flow collections (<a href="https://redirect.github.com/dtolnay/unsafe-libyaml/issues/26">dtolnay/unsafe-libyaml#26</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/serde-yaml/commit/f4c9ed92385c827a677dae60c2b5a894e24709f0"><code>f4c9ed9</code></a> Release 0.9.33</li> <li><a href="https://github.com/dtolnay/serde-yaml/commit/b4edaee907373ee69c6389687a35fced63d8addf"><code>b4edaee</code></a> Pull in yaml_parser_fetch_more_tokens fix from libyaml</li> <li><a href="https://github.com/dtolnay/serde-yaml/commit/8a5542ced61ae21d0772e504fac01bd1dbfaaa6b"><code>8a5542c</code></a> Resolve non_local_definitions warning in test</li> <li>See full diff in <a href="https://github.com/dtolnay/serde-yaml/compare/0.9.32...0.9.33">compare view</a></li> </ul> </details> <br /> Updates `syn` from 2.0.52 to 2.0.53 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/syn/releases">syn's releases</a>.</em></p> <blockquote> <h2>2.0.53</h2> <ul> <li>Implement Copy, Clone, and ToTokens for syn::parse::Nothing (<a href="https://redirect.github.com/dtolnay/syn/issues/1597">#1597</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/syn/commit/32dcf8df303c7181d95325f159ffbb1870a73e70"><code>32dcf8d</code></a> Release 2.0.53</li> <li><a href="https://github.com/dtolnay/syn/commit/fd1f2aa9979c557e5b95c85f35b8949b8059153f"><code>fd1f2aa</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/syn/issues/1597">#1597</a> from dtolnay/copyprintnothing</li> <li><a href="https://github.com/dtolnay/syn/commit/4f6c0528d63874495ce23d197a898cd63cb04d60"><code>4f6c052</code></a> Implement ToTokens for syn::parse::Nothing</li> <li><a href="https://github.com/dtolnay/syn/commit/3f37543794e5d43952399847a2ecbc1eb39f609f"><code>3f37543</code></a> Implement Copy for syn::parse::Nothing</li> <li><a href="https://github.com/dtolnay/syn/commit/36a412217d5217fab5657ab03ebcf29102d87a64"><code>36a4122</code></a> Update test suite to nightly-2024-03-16</li> <li><a href="https://github.com/dtolnay/syn/commit/bd931069f5adb81168e7e6186ead0663cc605257"><code>bd93106</code></a> Revert "Temporarily disable nightly testing due to libLLVM link issue"</li> <li><a href="https://github.com/dtolnay/syn/commit/06166a77b7d465b0e4c45e1a2162f15031d7f722"><code>06166a7</code></a> Update test suite to nightly-2024-03-09</li> <li><a href="https://github.com/dtolnay/syn/commit/ed545e75d8fdcab5078789ad25fc0efadc14e59b"><code>ed545e7</code></a> Work around doc_markdown lint in test_precedence</li> <li><a href="https://github.com/dtolnay/syn/commit/7aef1edbba28afe1bb6f9ddc0b5095dcd5135849"><code>7aef1ed</code></a> Temporarily disable nightly testing due to libLLVM link issue</li> <li><a href="https://github.com/dtolnay/syn/commit/556b10bf201264172e4b840f8cf1be38b8d75738"><code>556b10b</code></a> Update test suite to nightly-2024-03-06</li> <li>Additional commits viewable in <a href="https://github.com/dtolnay/syn/compare/2.0.52...2.0.53">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Currently the xcm-executor returns an `Unimplemented` error if it receives any HRMP-related instruction. What I propose here, which is what we are currently doing in our forked executor at polimec, is to introduce a trait implemented by the executor which will handle those instructions. This way, if parachains want to keep the default behavior, they just use `()` and it will return unimplemented, but they can also implement their own logic to establish HRMP channels with other chains in an automated fashion, without requiring to go through governance. Our implementation is mentioned in the [polkadot HRMP docs](https://arc.net/l/quote/hduiivbu), and it was suggested to us to submit a PR to add these changes to polkadot-sdk. --------- Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: command-bot <>

Closes: #3704

* master: Make `availability-recovery-regression-bench` a benchmark (#3741) Add HRMP notification handlers to the xcm-executor (#3696) Bump the known_good_semver group with 2 updates (#3726) removed `pallet::getter` usage from cumulus pallets (#3471)

paritytech-cicd-pr · 2024-03-19T11:12:52Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5569259

sandreim

Thanks @ordian !

alindima

Looking good

On top of paritytech#3302. We want the validators to upgrade first before we add changes to the collation side to send the new variants, which is why this part is extracted into a separate PR. The detection of when to send the parent head is based on the core assignments at the relay parent of the candidate. We probably want to make it more flexible in the future, but for now, it will work for a simple use case when a para always has multiple cores assigned to it. --------- Signed-off-by: Matteo Muraca <mmuraca247@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Matteo Muraca <56828990+muraca@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ignacio Rios <54085674+JuaniRios@users.noreply.github.com> Co-authored-by: Branislav Kontur <bkontur@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de>

ordian added 12 commits February 13, 2024 09:38

add new collation advertisement

504b919

remove v3 in favor of enum variant

efceb5e

get rid of with_elastic_scaling

94ba49b

no maybes

211edb7

prdoc

3320447

sanity check the provided parent head data hash

10a3e9c

address review: use enum for parent head-data

b7466c2

test validator sanity check parent_head_data hash

1ff7a54

collator-side: send parent head data

bd2ee35

When elastic scaling is detected (at the relay parent state), meaning more than one is assigned to the para at the time, the collator will send the collation variant with parent head data.

test parent head data is sent

560da60

ordian mentioned this pull request Feb 29, 2024

collator protocol changes for elastic scaling (validator side) #3302

Merged

2 tasks

ordian marked this pull request as ready for review February 29, 2024 11:08

sandreim reviewed Feb 29, 2024

View reviewed changes

Comment thread polkadot/node/network/collator-protocol/src/collator_side/mod.rs Outdated

Comment thread polkadot/node/network/collator-protocol/src/collator_side/mod.rs Outdated

ordian added 2 commits March 1, 2024 03:06

address nits

a7cc059

fmt

cc5f0e5

ordian added the T9-cumulus This PR/Issue is related to cumulus. label Mar 1, 2024

ordian added 3 commits March 1, 2024 03:18

prdoc

e2751f5

sandreim approved these changes Mar 1, 2024

View reviewed changes

AndreiEres approved these changes Mar 1, 2024

View reviewed changes

ordian added 3 commits March 4, 2024 08:59

add a comment (review)

7322058

Base automatically changed from ao-collator-parent-head-data to master March 15, 2024 20:06

ordian linked an issue Mar 16, 2024 that may be closed by this pull request

collator protocol: Provide parent head data #3128

Closed

ordian mentioned this pull request Mar 16, 2024

collator protocol: Provide parent head data #3128

Closed

ordian marked this pull request as draft March 18, 2024 13:22

Merge branch 'master' into ao-send-parent-head-data

1e2a32c

sandreim reviewed Mar 19, 2024

View reviewed changes

Comment thread polkadot/node/network/collator-protocol/src/collator_side/mod.rs

muraca and others added 7 commits March 19, 2024 11:54

removed pallet::getter usage from cumulus pallets (#3471)

4f4ef35

Part of #3326 @ggwpez @kianenigma @shawntabrizi polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp --------- Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>

Make availability-recovery-regression-bench a benchmark (#3741)

4697b7b

Closes: #3704

gate collator changes under elastic-scaling-experimental

d40a2a6

taplo format --config .config/taplo.toml

ee7ccdf

fix clippy?

2579796

ordian marked this pull request as ready for review March 19, 2024 13:17

sandreim approved these changes Mar 19, 2024

View reviewed changes

alindima approved these changes Mar 19, 2024

View reviewed changes

sandreim added this pull request to the merge queue Mar 19, 2024

Merged via the queue into master with commit 5fd72a1 Mar 19, 2024

sandreim deleted the ao-send-parent-head-data branch March 19, 2024 15:23

TDemeco mentioned this pull request Sep 27, 2024

feat: ⏫ upgrade to Polkadot SDK v1.10.0 Moonsong-Labs/storage-hub#210

Merged

alindima mentioned this pull request Feb 26, 2025

collator protocol: always send full parent head data in collation response #7733

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

collator-side: send parent head data#3521

collator-side: send parent head data#3521
sandreim merged 30 commits into
masterfrom
ao-send-parent-head-data

ordian commented Feb 29, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

sandreim left a comment

Uh oh!

alindima commented Mar 18, 2024

Uh oh!

sandreim left a comment

Uh oh!

Uh oh!

paritytech-cicd-pr commented Mar 19, 2024

Uh oh!

sandreim left a comment

Uh oh!

alindima left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

ordian commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sandreim left a comment

Choose a reason for hiding this comment

Uh oh!

alindima commented Mar 18, 2024

Uh oh!

sandreim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

paritytech-cicd-pr commented Mar 19, 2024

Uh oh!

sandreim left a comment

Choose a reason for hiding this comment

Uh oh!

alindima left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ordian commented Feb 29, 2024 •

edited

Loading