This release contains the changes from polkadot-stable2409-2 to polkadot-stable2412.

ℹ️ Please note:
The tag corresponding to the current stable release polkadot-stable2412 and matching the old pattern will be available under polkadot-v1.17.0.

This release contains two additions comparing to the previous releases:

From this release, in addition to the Linux binaries, binaries compiled for MacOS are available for:

polkadot
polkadot-execute-worker
polkadot-prepare-worker
polkadot-parachain
polkadot-omni-node

gpg signature has changed slightly, all the binaries are signed with the new release team key: "ParityReleases release-team@parity.io", which is part of the keyring as well as the previous key: "Parity Security Team security@parity.io".

Changelog

Changelog for `Node Dev`

ℹ️ These changes are relevant to: Those who build around the client side code. Alternative client builders, SMOLDOT, those who consume RPCs. These are people who are oblivious to the runtime changes. They only care about the meta-protocol, not the protocol itself.

[#5883]: statement-distribution RFC103 implementation

Introduces checks for the new candidate descriptor fields: core_index and session_index.

[#6864]: Fix approval-voting canonicalize off by one

The approval-voting canonicalize was off by one, which lead to blocks being cleaned up every other 2 blocks. Normally, this is not an issue, but on restart we might end up sending NewBlocks to approval-distribution with finalized blocks.
This would be problematic in the case were finalization was already lagging before restart, so after restart approval-distribution will trigger aggression on the wrong already finalized block.

[#5919]: substrate-offchain: upgrade hyper to v1

Bump depencency hyper of substrait-offchain for http from 0.14 to 1.
This changed APIs a bit;

sc_offchain::Offchainworker::new() now returns std::io::Result<Self> (Previously was Self)

[#5908]: collation-generation: use v2 receipts

Implementation of RFC 103 for the collation-generation subsystem.
Also removes the usage of AsyncBackingParams.

[#6729]: Fix order of resending messages after restart

At restart when dealing with a coalesced approval we might end up in a situation where we sent to approval-distribution the approval before all assignments covering it, in that case, the approval is ignored and never distribute, which will lead to no-shows.

[#5372]: elastic scaling: add core selector to cumulus

Adds a runtime API for querying the core selector of a parachain.
Also use the core selector API and the claim queue relay chain runtime API in the slot based collator (if available) to determine which cores to build on.
Part of implementing polkadot-fellows/RFCs#103.

[#5768]: export NodeHealthProxyLayer

This PR export NodeHealthProxyLayer from sc-rpc-server.

[#5911]: Removed the possibility to start a shell parachain node

Removed the possibility to start a shell parachain node using the polkadot-parachain-lib or polkadot-parachain-bin.

[#6760]: chainHead: Always report discarded items for storage operations

This PR ensures that substrate always reports discarded items as zero.
This is needed to align with the rpc-v2 spec

[#5515]: Add retry logic in relay chain rpc interface

Added a basic retry logic for collators connecting to external RPC servers. The collator will try for 5 times to connect to each RPC server from the provided list. In between each iteration will wait a duration which will increase exponentailly by a factor of two.
The maximum time a collator can spend in the retry logic is 1 + 2 + 4 + 8 + 16 = 31 seconds.

[#5540]: Avoid unnecessary block gap updates

Previously, the block gap storage in database and state in BlockchainDb could be updated even if no changes occurred.
This commit refines the logic to ensure updates only occur when the block gap value actually changes, reducing unnecessary writes and enhancing overall efficiency.

[#5635]: Fix edge case where state sync is not triggered

There is an edge case where the finalized block notification is received, but the conditions required to initiate the state sync are not fully met. In such cases, state sync would fail to start as expected and remain stalled.
This patch addresses it by storing the pending attempt and trying to start the state sync later when the conditions are satisfied.

[#5469]: Syncing strategy refactoring

Mostly internal changes to syncing strategies that is a step towards making them configurable/extensible in the future. It is unlikely that external developers will need to change their code.

[#5774]: Avoid unnecessary state reset of allowed_requests when no block requests are sent

Previously, the state of allowed_requests was always reset to the default even if there were no new block requests. This could cause an edge case because peer_block_request() will return early next time when there are no ongoing block requests.
This patch fixes it by checking whether block requests are empty before updating the state.

[#5038]: Plumb RPC listener up to caller

This PR allows the RPC server's socket address to be returned when initializing the server. This allows the library consumer to easily programmatically determine which port the RPC server is listening on.

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

Introduces a new trait TransactionExtension to replace SignedExtension. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have General for the former and Bare for the latter.
Unsigned will be phased out as a type of transaction, and Bare will only be used for Inherents.

Types of extrinsic are now therefore

Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
- Bare transactions (deprecated) - Gossiped, validated with ValidateUnsigned
  (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
- Inherents - Not gossiped, validated with ProvideInherent.
Extended (Extra data) - Gossiped, validated via TransactionExtension.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).

Notable information on TransactionExtension and the differences from SignedExtension

AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate.
pre_dispatch is renamed to prepare.
validate runs transaction validation logic both off-chain and on-chain, and is non-mutating.
prepare runs on-chain pre-execution logic using information extracted during validation and is mutating.
validate and prepare are now passed an Origin rather than an AccountId. If the extension logic presumes an AccountId, consider using the trait function AsSystemOriginSigner::as_system_origin_signer.
A signature on the underlying transaction may validly not be present.
The origin may be altered during validation.
Validation functionality present in validate should not be repeated in prepare.
Useful information obtained during validate should now be passed in to prepare using the new user-specifiable type Val.
Unsigned logic should be temporarily migrated from the old *_unsigned functions into the regular versions of the new functions where the Origin is None, until the deprecation of ValidateUnsigned in phase 2 of Extrinsic Horizon.
The Call type defining the runtime call is now a type parameter.
Extensions now track the weight they consume during validation, preparation and post-dispatch through the TransactionExtensionBase::weight function.
TestXt was removed and its usage in tests was replaced with UncheckedExtrinsic instances.

To fix the build issues introduced by this change, use the AsTransactionExtension adapter to wrap existing SignedExtensions by converting them using the From<SignedExtension> generic implementation for AsTransactionExtension. More details on migrating existing SignedExtension implementations to TransactionExtension in the PR description.

[#4837]: Add PVF execution priority

The new logic optimizes the distribution of execution jobs for disputes, approvals, and backings.
The main goal is to create back pressure for backing in the presence of disputes or numerous approval jobs.

[#5521]: Allow to call arbitrary runtime apis using RelayChainInterface

This PR adds a call_runtime_api method to RelayChainInterface trait, and a separate function also named call_runtime_apiwhich allows the caller to specify the input and output types, as opposed to having to encode them.

[#5686]: sync: Remove checking of the extrinsics root

Remove checking the extrinsics root as part of the sync code.
With the introduction of system_version and the possibility to use the V1 layout for the trie when calculating the extrinsics root, it would require the sync code to fetch the runtime version first before knowing which layout to use when building the extrinsic root.
The extrinsics root is still checked when executing a block on chain.

[#5601]: Introduce `RpcParams` in sc-cli

Refactors and consolidates all RPC-related parameters in the run command into a dedicated RpcParams struct. This change allows downstream users to build custom run command without duplicating code.

[#4849]: Introduce approval-voting-parallel subsystem

This introduces a new subsystem called approval-voting-parallel. It combines the tasks previously handled by the approval-voting and approval-distribution subsystems.

The new subsystem is enabled by default on all test networks. On production networks
like Polkadot and Kusama, the legacy system with two separate subsystems is still in use.
However, there is a CLI option --enable-approval-voting-parallel to gradually roll out the new subsystem on specific nodes. Once we are confident that it works as expected, it will be enabled by default on all networks.

The approval-voting-parallel subsystem coordinates two groups of workers:

Four approval-distribution workers that operate in parallel, each handling tasks based on the validator_index of the message originator.
One approval-voting worker that performs the tasks previously managed by the standalone approval-voting subsystem.

[#5830]: Remove jaeger from approval-voting and approval-distribution

Jaeger was remove from approval-voting and approval-distribution because it did not prove to improve the debugging and it wasted precious cpu cycles.

[#6603]: Always provide main protocol name in litep2p responses

This PR aligns litep2p behavior with libp2p. Previously, litep2p network backend would provide the actual negotiated request-response protocol that produced a response message. After this PR, only the main protocol name is reported to other subsystems.

[#5915]: Omni-Node renamings

This PR renames the polkadot-parachain-lib crate to polkadot-omni-node-lib and introduces a new polkadot-omni-node binary.

[#4974]: Remove libp2p dependency from sc-network-sync

This PR removes libp2p::request_response::OutboundFailure from substrate/client/network/sync/src/engine.rs.

[#5461]: runtime: remove ttl

Resolves #4776. Removes the scheduling ttl used in the relay chain runtimes, as well as the availability timeout retries. The extrinsics for configuring these two values are also removed.
Deprecates the ttl and max_availability_timeouts fields of the HostConfiguration primitive.

[#6676]: Expose the unstable metadata v16

This PR exposes the unstable metadata V16. The metadata is exposed under the unstable u32::MAX number.
Developers can start experimenting with the new features of the metadata v16. Please note that this metadata is under development and expect breaking changes until stabilization.
The ExtrinsicMetadata trait receives a breaking change. Its associated type VERSION is rename to VERSIONS and now supports a constant static list of metadata versions.
The versions implemented for UncheckedExtrinsic are v4 (legacy version) and v5 (new version).
For metadata collection, it is assumed that all TransactionExtensions are under version 0.

[#6218]: Enable approval-voting-parallel by default on kusama

Enable approval-voting-parallel by default on kusama

[#6268]: Bump a timeout in zombienet coretime smoke test

polkadot/zombienet_tests/smoke/0004-coretime-smoke-test.zndsl still timeouts on CI from time to time. Bumping the timeout a bit more.

Related to #6226

[#5616]: PVF: drop backing jobs if it is too late

Introduces the removal of backing jobs that have been back pressured for longer than allowedAncestryLen, as these candidates are no longer viable.

[#6156]: Use bool::then instead of then_some with function calls

Fix misusage of bool::then_some.

[#5998]: Fix memory leak in litep2p public addresses

This PR bounds the number of public addresses of litep2p to 32 entries.
This ensures we do not increase the number of addresses over time, and that the DHT authority records will not exceed the upper size limit.

[#5824]: Bump parachains runtime API to v11

This PR promotes all staging methods in v10 to stable and releases v11 stable runtime APIs.

[#6527]: Update litep2p network backend to version 0.8.1

Release 0.8.1 of litep2p includes critical fixes to further enhance the stability and performance of the litep2p network backend.

[#6646]: OmniNode --dev flag starts node with manual seal

polkadot-omni-node lib supports --dev flag now by allowing also to pass over a chain spec, and starts the node with manual seal. It will seal the node at each dev_block_time milliseconds, which can be set via --dev-block-time, and if not set will default to 3000ms.

[#5676]: [ci] Update CI image with rust 1.81.0 and 2024-09-11

cc https://github.com/paritytech/ci_cd/issues/1035

close https://github.com/paritytech/ci_cd/issues/1023

[#5859]: Add number of live peers available for requests

This PR adds a new metric for the number of live peers available for beefy requests.
The metric is exposed under the name substrate_beefy_on_demand_live_peers.

[#6015]: Rename QueueEvent::StartWork

When we send QueueEvent::StartWork, we have already completed the execution. Therefore, QueueEvent::FinishWork is a better match.

[#5732]: Expose the unstable metadata v16

[#4639]: Added the fork-aware transaction pool implementation

Most important changes introduced by this PR:

The transaction pool references spread across codebase are now wrapper to a transaction pool trait object,
The fork-aware pool implementation was added.
The sc-transaction-pool refactored,
Trasnaction pool builder was introduced to allow to instantiation of either old or new transaction pool. Refer to PR description for
more details on how to enable fork-aware pool in the custom node.

[#5924]: Bump PoV request timeout

With asynchronous backing and PoV size 10MB, we can increase the PoV request timeout from 1.2s to 2s.

[#5847]: `candidate-validation`: RFC103 implementation

Introduces support for new v2 descriptor core_index and session_index fields.
The subsystem will check the values of the new fields only during backing validations.

[#6677]: chore: Update litep2p to v0.8.2

This includes a critical fix for debug release versions of litep2p (which are running in Kusama as validators).

While at it, have stopped the oncall pain of alerts around incoming_connections_total. We can rethink the metric expose of litep2p in Q1.

[#5592]: Introduce `BlockGap`

This is the first step towards #5406, refactoring the representation of block gap. This refactor converts the existing (NumberFor<Block>, NumberFor<Block>) into a dedicated BlockGap<NumberFor<Block>>
struct. This change is purely structural and does not alter existing logic, but lays the groundwork for the follow-up PR. The compatibility concern in the database caused by the new structure transition is addressed as well.

The BlockGap refactoring results in breaking changes in the Info structure returned in client.info().

[#6353]: Update litep2p network backend to version 0.8.0

Release 0.8.0 of litep2p includes several improvements and memory leak fixes enhancing the stability and performance of the litep2p network backend.

[#6690]: Fix Possible bug, Vote import failed after aggression is enabled

Fix the appearance of Possible bug: Vote import failed after aggression is enabled, the log itself is harmless because approval gets imported anyway and aggression is able to distribute it, nevertheless is something that can be easily be fixed by picking the highest required routing possible.

[#6652]: rpc server: re-use server builder per rpc interface

This changes that the RPC server builder is re-used for each RPC interface which is more efficient than to build it for every connection.

[#5856]: Extend state tracking of chainHead to capture notification gaps

This PR extends the state tracking of the RPC-v2 chainHead methods.
ChainHead tracks the reported blocks to detect notification gaps.
This state tracking ensures we can detect NewBlock events for which we did not report previously the parent hash.

[#6742]: Update litep2p backend to v0.8.3

This release includes two fixes for small memory leaks on edge-cases in the notification and request-response protocols.
While at it, have downgraded a log message from litep2p.

[#6380]: Do not propagate external addr with different peerIDs

External addresses that belong to a different peerID are no longer propagated to the higher layers of the networking backends.

[#6217]: Unify and harden UMP signal checks in check_core_index

Refactors and hardens the core index checks on the candidate commitments.
Also adds a utility for skipping the ump signals

[#5880]: Fix prospective parachains test to use shuffled candidate list

Fix prospective parachains test to use shuffled candidate list.
Resolves #5617.

[#5997]: Implement archive_unstable_storageDiff method

This PR implements the archive_unstable_storageDiff rpc-v2 method.
Developers can use this method to fetch the storage differences
between two blocks. This is useful for oracles and archive nodes.
For more details see: https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/archive_unstable_storageDiff.md.

[#6298]: Populate authority DHT records with public listen addresses

This PR populates the authority DHT records with public listen addresses if any.
The change effectively ensures that addresses are added to the DHT record in the following order:

Public addresses provided by CLI --public-addresses
Maximum of 4 public (global) listen addresses (if any)
Any external addresses discovered from the network (ie from /identify protocol)

While at it, this PR adds the following constraints on the number of addresses:

Total number of addresses cached is bounded at 16 (increased from 10).
A maximum number of 32 addresses are published to DHT records (previously unbounded).
A maximum of 4 global listen addresses are utilized.

This PR replaces the following warning:
WARNING: No public address specified, validator node may not be reachable.
with a more descriptive one originated from the authority-discovery mechanism itself: No public addresses configured and no global listen addresses found.

[#5737]: Make syncing service an argument of `build_network`

build_network is accompanied with lower-level build_network_advanced with simpler API that does not create syncing engine internally, but instead takes a handle to syncing service as an argument. In most cases typical syncing engine with polkadot syncing strategy and default block downloader can be created with newly introduced sc_service::build_default_syncing_engine() function, but lower-level build_default_block_downloader also exists for those needing more customization.

These changes allow developers higher than ever control over syncing implementation, but build_network is still available for easier high-level usage.

[#5716]: Replace `lazy_static` with `LazyLock`

Replace all lazy_static usages with LazyLock from the Rust standard library. This will bring us less dependencies.

[#5679]: Switch to new `CandidateReceipt` primitives

This change is just plumbing work and updates all crate interfaces to use the new primitives.
It doesn't alter any functionality and is required before implementing RFC103 on the node side.

[#5789]: Prevents EthereumBlobExporter from consuming parameters when returning NotApplicable

When the EthereumBlobExporter returned a NotApplicable error, it consumed parameters universal_source, destination and message. As a result, subsequent exporters could not use these values. This PR corrects this incorrect behaviour. It also changes error type from Unroutable to NotApplicable when the global consensus system cannot be extracted from the universal_source, or when the source location cannot be converted to an agent ID. Lastly, it changes the error type from MissingArgument to NotApplicable when the parachain ID cannot be extracted from the location. These changes should have no effect - it is purely to correct behvaiour should multiple exporters be used.

[#5839]: Remove internal workaround for compiler bug

Remove a workaround we had in the impl_runtime_apis macro for a compiler bug that has been long fixed.
No impact on downstream users is expected, except relaxed trait bounds in a few places where the compiler is now able to deduce more type info itself.

[#5423]: Runtime support for candidate receipt v2 (RFC103)

Implementation of RFC103 in the relay chain runtime.
The runtime will accept and validate the new receipts only if the FeatureIndex::CandidateReceiptV2 feature bit is enabled.

[#6588]: rpc server: fix subscription id_provider being reset to default one

The modification ensures that the id_provider variable is cloned instead of taken, which can help prevent issues related id provider being reset to the default.

[#6016]: Litep2p network backend do not disconnect all peers on SetReservedPeers command

Previously, when the SetReservedPeers was received, all peers except the new reserved peers were disconnected.
This PR ensures that previously reserved nodes are kept connected as regular nodes if enough slots are available.
While at it, this PR excludes reserved peers from the candidates of peers obtained from the peerstore.

[#6382]: gensis-config: patching default `RuntimeGenesisConfig` fixed

This PR fixes issue reported in #6306.
It changes the behavior of sc_chain_spec::json_patch::merge function which no longer removes any keys from the base JSON object.

[#5787]: Move bitfield_distribution to blocking task pool and set capacity to 8192

This is moving bitfield_distribution to the blocking task pool because it does cpu intensive work and to make it snappier. Additionally, also increase the message capacity of the subsystem to make sure the queue does not get full if there is a burst of messages.

[#5762]: Fast return for invalid request of node health

Return directly when invalid request for node health api

[#5954]: templates: make node compilation optional

Node compilation for minimal and parachain templates is made optional, not part of the templates default-members list. At the same time, we introduce OmniNode as alternative to run the templates.

[#6406]: make prospective-parachains debug logs less spammy

Demote some of the frequent prospective-parachains debug logs to trace level and prefer printing aggregate debug logs.

[#5572]: added RPC metrics for the collator

The metric is named relay_chain_rpc_interface and can be scraped by prometheus agents from the parachain prometheus exporter. The metric provide information about count, sum and duration in seconds (with exponential buckets with parameters as start = 0.001, factor = 4, count = 9) for all RPC requests made with the relay-chain-rpc-interface.

[#6696]: Make approval-distribution aggression a bit more robust and less spammy

The problem with the current implementation of approval-distribution aggression is that is too spammy, and can overload the nodes, so make it less spammy by moving back the moment we trigger L2 aggression and make resend enable only for the latest unfinalized block.

[#6860]: Update litep2p network backend to v0.8.4

This PR updates the Litep2p network backend to version 0.8.4

[#6863]: Update merkleized-metadata to 0.2.0

0.1.2 was yanked as it was breaking semver.

[#5811]: Improve `import_notification_stream` documentation

"Updates the doc comment on the import_notification_stream to make its behaviour clearer. Now it specifically states that this notification stream is fired on every import notification after the initial sync, and only when there are re-orgs in the initial sync."

[#5343]: Allow to disable gap creation during block import

New property BlockImportParams::create_gap allows to change whether to create block gap in case block has no parent (defaults to true keeping existing behavior), which is helpful for sync protocols that do not need to sync the gap after this happens. BlockImportOperation::create_gap() method was also introduced, though in most cases BlockImportParams::create_gap will be used.

[#5322]: Elastic scaling - introduce new candidate receipt primitive

Introduces CandidateDescriptorV2 primitive as described in RFC 103.
Updates parachains runtime, Westend, Rococo and test runtimes to use the new primitives.
This change does not implement the functionality of the new candidate receipts.

[#5609]: Update litep2p network backend to v0.7.0

This PR updates the Litep2p network backend to version 0.7.0.
This new release introduces several new features, improvements, and fixes to the litep2p library.
Key updates include enhanced error handling propagated through metrics, configurable connection limits, and a new API for managing public addresses.

The Identify protocol no longer includes public addresses in its configuration.
Instead, we rely on the litep2p.public_addresses interface to propagate external addresses of the node.

Litep2p uses hickory DNS resolver (formerly known as trust DNS).
Similarly to the trust DNS, the hickory logs are silenced.

[#6104]: LocalTransactionPool implemented for fork aware transaction pool

LocalTransactionPool trait is implemented for fork aware transaction pool.

[#5666]: Make syncing strategy an argument of the syncing engine

Syncing strategy is no longer implicitly created when building network, but needs to be instantiated explicitly.
Previously default implementation can be created with new function build_polkadot_syncing_strategy or custom syncing strategy could be implemented and used instead if desired, providing greater flexibility for chain developers.

[#4846]: Make approval-voting runnable on a worker thread

Make approval-voting subsystem runnable on a separate worker thread without having to to always pass to it an orchestra context. It achieves that by refactoring existing functions to require only the minimal set of traits needed in the function instead of the general
Context

[#5707]: Remove ValidateFromChainState

Removed the CandidateValidationMessage::ValidateFromChainState, which was previously used by backing, but is no longer relevant since initial async backing implementation

[#6011]: collator protocol: validate descriptor version on the validator side

Implement checks needed for RFC 103 polkadot-fellows/RFCs#103 in the validator side of the collator protocol.

[#5875]: Remove jaeger from polkadot

Jaeger was remove from the codebase because it was not used by anyone and it did not help with the debugging.

Changelog for `Runtime Dev`

ℹ️ These changes are relevant to: All of those who rely on the runtime. A parachain team that is using a pallet. A DApp that is using a pallet. These are people who care about the protocol (WASM, not the meta-protocol (client).)

[#5961]: Bounties Pallet: add `approve_bounty_with_curator` call

Adds approve_bounty_with_curator call to the bounties pallet to combine approve_bounty and propose_curator into one call. If unassign_curator is called after approve_bounty_with_curator the process falls back to the previous flow of calling propose_curator separately. Introduces a new ApprovedWithCurator bounty status when bounty is approved with curator.

[#4851]: Add support for deprecation metadata in `RuntimeMetadataIr` entries.

Changes introduced are listed below.
Adds DeprecationStatusIR enum to sp_metadata_ir.

Is a deprecation info for simple items.
Adds DeprecationInfoIR enum to sp_metadata_ir.
It is a deprecation info for an enums/errors/calls. Contains DeprecationStatusIR.
Also denotes full/partial deprecation of the type or its variants/calls.
Adds deprecation_info field to
RuntimeApiMetadataIR
RuntimeApiMethodMetadataIR
StorageEntryMetadataIR
PalletConstantMetadataIR
PalletCallMetadataIR
PalletMetadataIR
PalletEventMetadataIR
PalletErrorMetadataIR
Examples of the deprecation info produced can be seen inside
Tests for frame-support
hackmd link https://hackmd.io/@Zett98/Bys0YgbcR

[#6418]: Follow up work on TransactionExtension - fix weights and clean up UncheckedExtrinsic

This PR removes the redundant extension version byte from the signed v4 extrinsic, previously unused and defaulted to 0. The extension version byte is also made to be part of the inherited implication handed to extensions in General transactions. Also, some system extensions benchmarks were adjusted through whitelisting to not count the reads for frequently read storage keys.

[#6316]: Migrate pallet-election-provider-multi-phase benchmark to v2 and improve doc

Migrate pallet-election-provider-multi-phase benchmark to v2 and improve doc

[#5876]: (XCMv5) implement RFC#100, add new InitiateTransfer instruction

There's a new XCM instruction in v5: InitiateTransfer.
It's meant as a general instruction that will do everything (and more) currently done by InitiateTeleport, InitiateReserveWithdraw and DepositReserveAsset.
Its main feature is the ability to do cross-chains transfers mixing teleported and reserve transferred assets.

/// Specify which type of asset transfer is required for a particular `(asset, dest)` combination.
pub enum AssetTransferFilter {
	/// teleport assets matching `AssetFilter` to `dest`
	Teleport(AssetFilter),
	/// reserve-transfer assets matching `AssetFilter` to `dest`, using the local chain as reserve
	ReserveDeposit(AssetFilter),
	/// reserve-transfer assets matching `AssetFilter` to `dest`, using `dest` as reserve
	ReserveWithdraw(AssetFilter),
}
/// Cross-chain transfer matching `assets` in the holding register as follows:
///
/// Assets in the holding register are matched using the given list of `AssetTransferFilter`s,
/// they are then transferred based on their specified transfer type:
///
/// - teleport: burn local assets and append a `ReceiveTeleportedAsset` XCM instruction to
///   the XCM program to be sent onward to the `dest` location,
///
/// - reserve deposit: place assets under the ownership of `dest` within this consensus system
///   (i.e. its sovereign account), and append a `ReserveAssetDeposited` XCM instruction
///   to the XCM program to be sent onward to the `dest` location,
///
/// - reserve withdraw: burn local assets and append a `WithdrawAsset` XCM instruction
///   to the XCM program to be sent onward to the `dest` location,
///
/// The onward XCM is then appended a `ClearOrigin` to allow safe execution of any following
/// custom XCM instructions provided in `remote_xcm`.
///
/// The onward XCM also potentially contains a `BuyExecution` instruction based on the presence
/// of the `remote_fees` parameter (see below).
///
/// If a transfer requires going through multiple hops, an XCM program can compose this instruction
/// to be used at every chain along the path, describing that specific leg of the transfer.
///
/// Parameters:
/// - `dest`: The location of the transfer next hop.
/// - `remote_fees`: If set to `Some(asset_xfer_filter)`, the single asset matching
///   `asset_xfer_filter` in the holding register will be transferred first in the remote XCM
///   program, followed by a `BuyExecution(fee)`, then rest of transfers follow.
///   This guarantees `remote_xcm` will successfully pass a `AllowTopLevelPaidExecutionFrom` barrier.
/// - `remote_xcm`: Custom instructions that will be executed on the `dest` chain. Note that
///   these instructions will be executed after a `ClearOrigin` so their origin will be `None`.
///
/// Safety: No concerns.
///
/// Kind: *Command*.
///
InitiateTransfer {
	destination: Location,
	remote_fees: Option<AssetTransferFilter>,
	assets: Vec<AssetTransferFilter>,
	remote_xcm: Xcm<()>,
}

[#5194]: FRAME: Support instantiable pallets in tasks.

In FRAME, tasks can now be used in instantiable pallet. Also some fix for expansion with conditional compilation in construct runtine.

[#6032]: Fix `feeless_if` in pallet section

Fix compilation with pallet::feeless_if in a pallet section: a local binding unexpectely resolved to a macro definition.

[#6192]: [pallet-revive] fix hardcoded gas in tests

Fix hardcoded gas limits in tests

[#5917]: XCM paid execution barrier supports more origin altering instructions

Updates the AllowTopLevelPaidExecutionFrom barrier to also support messages that use DescendOrigin or AliasOrigin for altering the computed origin during execution.

[#3151]: Dynamic deposit based on number of proposals

Introduce a dynamic proposal deposit mechanism influenced by the total number of active proposals, with the option to set the deposit to none.

The potential cost (e.g., balance hold) for proposal submission and storage is determined by the implementation of the Consideration trait. The footprint is defined as proposal_count, representing the total number of active proposals in the system, excluding the one currently being proposed. This cost may vary based on the proposal count. The pallet also offers various types to define a cost strategy based on the number of proposals.

Two new calls are introduced:

kill(origin, proposal_hash): the cancellation of a proposal, accompanied by the burning of the associated cost/consideration ticket.
release_proposal_cost(origin, proposal_hash): the release of the cost for a non-active proposal.

New config parameters:

DisapproveOrigin: origin from which a proposal in any status may be disapproved without associated cost for a proposer;
KillOrigin: Origin from which any malicious proposal may be killed with associated cost for a proposer;
Consideration: mechanism to assess the necessity of some cost for publishing and storing a proposal. Set to unit type to have not submission cost;

Additionally change:

benchmarks have been upgraded to benchmarks::v2 for collective pallet;
ensure_successful function added to the Consideration under runtime-benchmarks feature.

[#5372]: elastic scaling: add core selector to cumulus

[#5665]: [pallet-contracts] remove riscv support

RISC-V support is now being built inside the new fork pallet-revive

[#6496]: Backport #6418 to `stable2412`

This PR is a backport of #6418.

For context, TransactionExtension, introduced in #3685, is part of the stable2412 release, and this PR brings important fixes and quality of life improvements. Doing the backport now allows us to not break the interface later.

Opened against #6473 as the changes in the original PR were made on top of the changes in this backport.

[#5891]: Add benchmark overhead command to frame-omni-bencher

This adds the benchmark overhead command to the frame-omni-bencher library. This allows para- and relay chain teams to generate extrinsic and block base weights.

[#5866]: [pallet-revive] Ethereum JSON-RPC integration

Changes Included:

A new pallet::call eth_transact.
A custom UncheckedExtrinsic struct to dispatch unsigned eth_transact calls from an Ethereum JSON-RPC proxy.
Generated types and traits to support implementing a JSON-RPC Ethereum proxy.

[#6645]: xcm: fix local/remote exports when inner routers return `NotApplicable`

Resolved a bug in the local/remote exporters used for bridging. Previously, they consumed dest and msg without returning them when inner routers/exporters failed with NotApplicable. This PR ensures compliance with the SendXcm and ExportXcm traits.

[#4826]: XCMv5

Added XCMv5.

This PR brings a new XCM version.
It's an amalgamation of multiple individual PRs:

XCMv5 reduces the potential for bugs by:

Removing the need to specify weight in Transact.
Handling fees in a better way with PayFees instead of BuyExecution.
Improves asset claiming with SetAssetClaimer.

It also allows some new use-cases like:

Sending both teleported and reserve asset transferred assets in the same cross-chain transfer.
Preserving the origin when doing cross-chain transfers. Allowing the use of Transact in the same message as a cross-chain transfer.

In version 5, it's expected to change usage of BuyExecution to PayFees.
While BuyExecution returns all leftover assets to holding, PayFees doesn't.
The only way to get funds back from those sent to PayFees is by using RefundSurplus.
Because of this, it's meant to be used alongside the new DryRunApi and XcmPaymentApi.
You first dry-run the XCM, get the fees needed, and put them in PayFees.

[#6025]: Refactor staking pallet benchmarks to `v2`

Update benchmarks in staking pallet to the second version of the frame_benchmarking runtime benchmarking framework.

[#6073]: Refactor `pallet-grandpa` benchmarks to `v2`

Update benchmarks in GRANDPA pallet to use the second version of the frame_benchmarking runtime benchmarking framework.

[#5640]: [pallet-revive] Move event's topics

Move event's topics inside body

[#6169]: [Deprecation] deprecate treasury `spend_local` call and related items

Deprecates spend_local from the treasury pallet and items associated with it.

Migration

For users who were using only `spend_local` before

To replace spend_local functionality configure Paymaster pallet configuration to be PayFromAccount and configure AssetKind to be () and use spend call instead.
This way spend call will function as deprecated spend_local.

Example:

impl pallet_treasury::Config for Runtime {
  ..
  type AssetKind = ();
  type Paymaster = PayFromAccount<Self::Currency, TreasuryAccount>;
  // convert balance 1:1 ratio with native currency
  type BalanceConverter = UnityAssetBalanceConversion;
  ..
}

For users who were already using `spend` with all other assets, except the native asset

Use NativeOrWithId type for AssetKind and have a UnionOf for native and non-native assets, then use that with PayAssetFromAccount.

Example from kitchensink-runtime:

// Union of native currency and assets
pub type NativeAndAssets =
  UnionOf<Balances, Assets, NativeFromLeft, NativeOrWithId<u32>, AccountId>;

impl pallet_treasury::Config for Runtime {
  ..
  type AssetKind = NativeOrWithId<u32>;
  type Paymaster = PayAssetFromAccount<NativeAndAssets, TreasuryAccount>;
  type BalanceConverter = AssetRate;
  ..
}

// AssetRate pallet configuration
impl pallet_asset_rate::Config for Runtime {
  ..
  type Currency = Balances;
  type AssetKind = NativeOrWithId<u32>;
  ..
}

[#6257]: fix claim queue size when validator groups count is smaller

Fixes a bug introduced in #5461, where the claim queue would contain entries even if the validator groups storage is empty (which happens during the first session). This PR sets the claim queue core count to be the minimum between the num_cores param and the number of validator groups.

[#5765]: Added foreign locations to local accounts converter to all the parachains.

Added foreign locations to local accounts converter to all the parachains.
i.e. added HashedDescription<AccountId, DescribeFamily> to LocationToAccountId

[#6171]: remove parachains_assigner

Remove the code of the parachains_assigner pallet, since coretime was released on all production networks.

[#6318]: Refactor pallet claims

Adds bounds on stored types for pallet claims.
Migrates benchmarking from v1 to v2 for pallet claims.

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

Types of extrinsic are now therefore

Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
- Bare transactions (deprecated) - Gossiped, validated with ValidateUnsigned (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
- Inherents - Not gossiped, validated with ProvideInherent.
Extended (Extra data) - Gossiped, validated via TransactionExtension.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).

Notable information on TransactionExtension and the differences from SignedExtension

AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate.
pre_dispatch is renamed to prepare.
validate runs transaction validation logic both off-chain and on-chain, and is non-mutating.
prepare runs on-chain pre-execution logic using information extracted during validation and is mutating.
validate and prepare are now passed an Origin rather than an AccountId. If the extension logic presumes an AccountId, consider using the trait function AsSystemOriginSigner::as_system_origin_signer.
A signature on the underlying transaction may validly not be present.
The origin may be altered during validation.
Validation functionality present in validate should not be repeated in prepare.
Useful information obtained during validate should now be passed in to prepare using the new user-specifiable type Val.
Unsigned logic should be temporarily migrated from the old *_unsigned functions into the regular versions of the new functions where the Origin is None, until the deprecation of ValidateUnsigned in phase 2 of Extrinsic Horizon.
The Call type defining the runtime call is now a type parameter.
Extensions now track the weight they consume during validation, preparation and post-dispatch through the TransactionExtensionBase::weight function.
TestXt was removed and its usage in tests was replaced with UncheckedExtrinsic
instances.

[#6087]: Expose private structs in pallet_nfts and pallet_uniques.

PR changes certain structs in pallet_nfts and pallet_uniques into public. It also changes 2 storages (collection & asset metadata) into public in pallet_uniques.

[#6264]: pallet-revive: Trade code size for call stack depth

This will reduce the call stack depth in order to raise the allowed code size. Should allow around 100KB of instructions. This is necessary to stay within the memory envelope. More code size is more appropriate for testing right now. We will re-evaluate parameters once we have 64bit support.

[#5726]: revive: Limit the amount of static memory

Limit the amount of static memory a contract can declare.

[#6505]: [pallet-broker] Fix auto renew benchmarks

Fix the broker pallet auto-renew benchmarks which have been broken since #4424, yielding Weightless due to some prices being set too low, as reported in #6474.

Upon further investigation it turned out that the auto-renew contribution to rotate_sale was always failing but the error was mapped. This is also fixed at the cost of a bit of setup overhead.

[#5548]: Use H160 when interfacing with contracts

When interfacing with a contract we now use the native ethereum address type and map it to AccountId32 when interfacing with the rest of substrate.

[#5971]: XCMv5 InitiateTransfer can preserve original origin across chains.

Beware: This only works if the following two rules are followed by the chain receiving such a message.

Alias to interior locations is valid (the exact same behaviour as DescendOrigin)
AssetHub can alias everything (most importantly sibling accounts and ethereum).
These can be set with the Aliasers configuration item, with the following adapters:
AliasChildLocation
AliasOriginRootUsingFilter with AssetHub and Everything
An example of the first one can be seen in asset-hub-westend and of the second one in penpal-runtime.

[#5585]: Added SetAssetClaimer instruction to XCM v5.

Added SetAssetClaimer implementation to XCM v5. With asset_claimer set users can retrieve their trapped assets at any point in time without the need to go through OpenGov reclaim process.

[#5608]: [pallet-revive] update runtime types

Refactor the Ext trait to use U256 instead of BalanceOf or MomentOf

[#6778]: Added fallback_max_weight to Transact for sending messages to V4 chains

Removing the require_weight_at_most parameter in V5 Transact introduced a problem when converting a message from V5 to V4 to send to chains that didn't upgrade yet.
The local chain doesn't know how to decode calls for remote chains so it can't automatically populate require_weight_at_most required by V4 Transact.
To fix this, XCM v5 Transact now also takes a fallback_max_weight: Option<Weight> parameter.
This can be set to None if the instruction is not meant to be sent to chains running XCM versions lower than V5.
If set to Some(weight), a subsequent conversion to V4 will result in Transact { require_weight_at_most: weight, .. }.
The plan is to remove this workaround in V6 since there will be a good conversion path from V6 to V5.

[#5201]: Snowbridge free consensus updates

Allow free consensus updates to the Snowbridge Ethereum client if the headers are more than a certain number of headers apart. Relayers providing valid consensus updates are refunded for updates. Bridge users are not affected.

[#5745]: Implement `try_append` for `StorageNMap`

This PR introduces the try_append api which is available on other storage map types, but missing on StorageNMap.

[#6317]: eth-rpc fixes

Various fixes for the release of eth-rpc & ah-westend-runtime:

Bump asset-hub westend spec version
Fix the status of the Receipt to properly report failed transactions
Fix value conversion between native and eth decimal representation

[#6295]: Migrate pallet-im-online benchmark to v2

Part of:

#6202.

[#6278]: [pallet-revive] rpc server add docker file

Add a docker for pallet-revive eth-rpc

Tested with

sudo docker build . -t eth-rpc -f substrate/frame/revive/rpc/Dockerfile
sudo docker run --network="host" -e RUST_LOG="info,eth-rpc=debug" eth-rpc

[#6260]: [pallet-revive] code size API

This PR implements the contract API to query the code size of a given address.

[#6045]: [pallet-revive] ensure the return data is reset if no frame was instantiated

Failed call frames do not produce new return data but still reset it.

[#6174]: [pallet-revive] fix fixture build path

Fix fixture build path

[#5461]: runtime: remove ttl

[#5888]: parachain-system: send core selector ump signal

Send the core selector ump signal in cumulus. Guarded by a compile time feature called experimental-ump-signals until nodes are upgraded to a version that includes #5423 for gracefully handling ump signals.

[#4012]: `impl_runtime_apis!`: replace the use of `Self` with `Runtime`

Currently, if there is a type alias similar to type HeaderFor<T> in the scope, it makes sense to expect that HeaderFor<Runtime> and HeaderFor<Self> are equivalent. However, this is not the case. It currently leads to a compilation error that Self is not in scope, which is confusing. This PR introduces a visitor, similar to CheckTraitDecl in decl_runtime_apis!, ReplaceSelfImpl. It identifies usage of Self as a type argument in impl_runtime_apis! and replaces Self with an explicit Runtime type.

For example, the following example code will be transformed before expansion:

impl apis::Core<Block> for Runtime {
    fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
        let _: HeaderFor<Self> = header.clone();
        RuntimeExecutive::initialize_block(header)
    }
}

Instead, it will be passed to macro as:

impl apis::Core<Block> for Runtime {
    fn initialize_block(header: &HeaderFor<Runtime>) -> ExtrinsicInclusionMode {
        let _: HeaderFor<Runtime> = header.clone();
        RuntimeExecutive::initialize_block(header)
    }
}

[#5556]: Make salt optional

By making salt optional we allow clients to use CREATE1 semantics when deploying a new contract.

[#6221]: snowbridge: allow account conversion for Ethereum accounts

Replaced GlobalConsensusEthereumConvertsFor with EthereumLocationsConverterFor that allows Location to AccountId conversion for the Ethereum network root as before, but also for Ethereum contracts and accounts.

[#6039]: Added Trusted Query API calls.

Added is_trusted_reserve and is_trusted_teleporter API calls to all the runtimes.
Given an asset and a location, they return if the chain trusts that location as a reserve or teleporter for that asset respectively.
You can implement them on your runtime by simply calling a helper function on pallet-xcm.

	impl xcm_runtime_apis::trusted_query::TrustedQueryApi<Block> for Runtime {
  fn is_trusted_reserve(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_reserve(asset, location)
  }
  fn is_trusted_teleporter(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_teleporter(asset, location)
  }
}

[#5526]: Fix enact_candidate weight generation

This PR works around an issue in multivariate linear regression of weight generation.

[#5274]: Enrich metadata IR with associated types of config traits

This feature is part of the upcoming metadata V16. The associated types of the Config trait that require the TypeInfo or Parameter bounds are included in the metadata of the pallet. The metadata is not yet exposed to the end-user, however the metadata intermediate representation (IR) contains these types.

Developers can opt out of metadata collection of the associated types by specifying without_metadata optional attribute to the #[pallet::config].

Furthermore, the without_metadata argument can be used in combination with the newly added #[pallet::include_metadata]
attribute to selectively include only certain associated types in the metadata collection.

[#5824]: Bump parachains runtime API to v11

This PR promotes all staging methods in v10 to stable and releases v11 stable runtime
APIs.

[#6315]: Migrate pallet-election-provider-support-benchmarking benchmark to v2

Migrate pallet-election-provider-support-benchmarking benchmark to v2

[#6246]: [pallet-revive] implement the block hash API

Bound T::Hash to H256
Implement the block hash API

[#6187]: [pallet-revive] rework balance transfers

This PR removes the transfer syscall and changes balance transfers to make the existential deposit (ED) fully transparent for contracts.

The transfer API is removed since there is no corresponding EVM opcode and transferring via a call introduces barely any overhead.

We make the ED transparent to contracts by transferring the ED from the call origin to nonexistent accounts. Without this change, transfers to nonexistant accounts will transfer the supplied value minus the ED from the contracts viewpoint, and consequentially fail if the supplied value lies below the ED. Changing this behavior removes the need for contract code to handle this rather annoying corner case and aligns better with the EVM. The EVM charges a similar deposit from the gas meter, so transferring the ED from the call origin is practically the same as the call origin pays for gas.

[#6646]: OmniNode --dev flag starts node with manual seal

[#6849]: Add aliasers to westend chains

InitiateTransfer, the new instruction introduced in XCMv5, allows preserving the origin after a cross-chain transfer via the usage of the AliasOrigin instruction. The receiving chain needs to be configured to allow such this instruction to have its intended effect and not just throw an error.

In this PR, I add the alias rules specified in the RFC for origin preservation to westend chains so we can test these scenarios in the testnet.

The new scenarios include:

Sending a cross-chain transfer from one system chain to another and doing a Transact on the same message (1 hop)
Sending a reserve asset transfer from one chain to another going through asset hub and doing Transact on the same message (2 hops)

The updated chains are:

Relay: added AliasChildLocation
Collectives: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>
People: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>
Coretime: added AliasChildLocation and AliasOriginRootUsingFilter<AssetHubLocation, Everything>

AssetHub already has AliasChildLocation and doesn't need the other config item.
BridgeHub is not intended to be used by end users so I didn't add any config item.
Only added AliasChildOrigin to the relay since we intend for it to be used less.

[#5676]: [ci] Update CI image with rust 1.81.0 and 2024-09-11

cc https://github.com/paritytech/ci_cd/issues/1035

close https://github.com/paritytech/ci_cd/issues/1023

[#5198]: MQ processor should be transactional

Enforce transactional processing on pallet Message Queue Processor.

Storage changes that were done while processing a message will now be rolled back when the processing returns an error. Ok(false) will not revert, only Err(_).

[#5502]: [pallet-revive] Add pallet to AH westend

Add pallet-revive to Westend runtime, and configure the runtime to accept Ethereum signed transaction

[#6305]: Remove `riscv` feature flag

Since #6266 we no longer require a custom toolchain to build the pallet-revive-fixtures. Hence we no longer have to guard the build behind a feature flag.

[#6228]: Transact without having to specify weight

In XCMv5, it's no longer required to pass in the expected weight when using Transact.
This was made to remove a whole class of bugs where the weight specified was not enough.

[#6088]: [pallet-revive] EXTCODEHASH to match EIP-1052

Update ext_code_hash to match EIP-1052 specs.

[#5390]: Remove NetworkIds for testnets Rococo and Westend

Implemetation of polkadot-fellows/RFCs#108, in the version 5 of XCM, Remove Westend and Rococo from the included NetworkIds to improve the stability of the language.

NetworkId::Rococo and NetworkId::Westend can just use NetworkId::ByGenesis by importing their genesis
block hash

[#6314]: Migrate pallet-elections-phragmen benchmark to v2 and improve doc

Part of:

#6202.

[#6288]: [pallet-revive] Add metrics to eth-rpc

Add metrics for eth-rpc

[#4251]: MBM `try-runtime` support

MBM try-runtime support

This MR adds support to the try-runtime trait such that the try-runtime-CLI will be able to support MBM testing here.
It mainly adds two feature-gated hooks to the SteppedMigration hook to facilitate testing. These hooks are named pre_upgrade and post_upgrade and have the same signature and implications as for single-block migrations.

Integration

To make use of this in your Multi-Block-Migration, just implement the two new hooks and test pre- and post-conditions in them:

#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, frame_support::sp_runtime::TryRuntimeError>
{
  // ...
}

#[cfg(feature = "try-runtime")]
fn post_upgrade(prev: Vec<u8>) -> Result<(), frame_support::sp_runtime::TryRuntimeError> {
    // ...
}

You may return an error or panic in these functions to indicate failure.
This will then show up in the try-runtime-CLI and can be used in CI for testing.

Changes:

Adds try-runtime gated methods pre_upgrade and post_upgrade on SteppedMigration
Adds try-runtime gated methods nth_pre_upgrade and nth_post_upgrade on SteppedMigrations
Modifies pallet_migrations implementation to run pre_upgrade and post_upgrade steps at the appropriate times, and panic in the event of migration failure.

[#6096]: pallet-revive: Add stateful address mapping

Fixes #5576

This allows contracts to be used with an AccountId32 through normal extrinsics and not only through the eth compat layer. It works by adding a new extrinsic map_account that lets people register their AccountId32.

[#5857]: Beefy equivocation: check all the MMR roots

This PR adjusts the logic for report_fork_voting exposed by pallet-beefy.
Normally, the BEEFY protocol only accepts a single MMR Root entry in a commitment's payload. But, in order to be extra careful, now, when validating equivocation reports, we check all the MMR roots, if there are more.

[#5743]: [pallet-revive] write sandbox output according to the provided output buffer length

Instead of error out if the provided output buffer is smaller than what we want to write, we can just write what fits into the output buffer instead.
We already write back the actual bytes written to the in-out pointer, so contracts can check it anyways.

This in turn introduces the benefit of allowing contracts to implicitly request only a portion of the returned data from calls and incantations.
Which is especially beneficial for YUL as the call family opcodes have a return data size argument and this change removes the need to work around it in contract code.

[#5941]: `SolochainDefaultConfig`: Use correct `AccountData`

SolochainDefaultConfig by default was setting AccountData to AccountInfo.
Thus, the actual account data was recursively nested the same type. By default it should be set (), because this is the only reasonable AccountData.

If you have used SolochainDefaultConfig before and did not overwrite, AccountData you should now overwrite it to AccountInfo or you will need to write a migration to change the data.

[#6255]: [pallet-child-bounties] Index child bounties by parent bounty

Index child bounties by their parent bounty, ensuring that their indexes are independent of child bounties from other parent bounties. This will allow for predictable indexes and the ability to batch creation and approval calls together.

Migration for Runtime Pallet Instance

Use migration::v1::MigrateToV1Impl storage migration type to translate ids for the active child bounties and migrate the state to the new schema.

Migration for Clients

Use new ParentTotalChildBounties storage item to iterate over child bounties for a certain parent bounty;
Use new ChildBountyDescriptionsV1 storage item to get the bounty description instead of removed ChildBountyDescriptions;
Use V0ToV1ChildBountyIds storage item to look up the new child bounty id for a given old child bounty id;
Update the child bounty account id derivation from PalletId + "cb" + child_id to PalletId + "cb" + bounty_id + child_id.

Additional Notes

The ChildBountyCount storage item is deprecated and will be remove in May 2025.

[#6263]: [pallet-revive] Update typeInfo

Update typeinfo impl to make it transparent for subxt

see paritytech/subxt#1845

[#6536]: Bridges testing improvements

This PR includes:

Refactored integrity tests to support standalone deployment of pallet-bridge-messages.
Refactored the open_and_close_bridge_works test case to support multiple scenarios, such as:
1. A local chain opening a bridge.
2. Sibling parachains opening a bridge.
3. The relay chain opening a bridge.
Previously, we added instance support for pallet-bridge-relayer but overlooked updating the DeliveryConfirmationPaymentsAdapter.

[#5813]: build_struct_json_patch macro added

This PR adds a macro that allows to construct a RuntimeGenesisConfig preset containing only provided fields, while performing the validation of the entire struct.

Related issue: #5700

[#5845]: Fix compilation after renaming some of benchmarks in pallet_revive.

Changed the "instr" benchmark so that it should no longer return to little weight. It is still bogus but at least benchmarking should not work.

[#5779]: [pallet-revive] last call return data API

This PR introduces 2 new syscall: return_data_size and return_data_copy, resembling the semantics of the EVM RETURNDATASIZE and RETURNDATACOPY opcodes.

The ownership of ExecReturnValue (the return data) has moved to the Frame.
This allows implementing the new contract API surface functionality in ext with no additional copies.
Returned data is passed via contract memory, memory is (will be) metered, hence the amount of returned data can not be statically known, so we should avoid storing copies of the returned data if we can.
By moving the ownership of the exectuables return value into the Frame struct we achieve this.

A zero-copy implementation of those APIs would be technically possible without that internal change by making the callsite in the runtime responsible for moving the returned data into the frame after any call.
However, resetting the stored output needs to be handled in ext, since plain transfers will not affect the stored return data (and we don't want to handle this special call case inside the runtime API).
This has drawbacks:

It can not be tested easily in the mock.
It introduces an inconsistency where resetting the stored output is handled in ext, but the runtime API is responsible to store it back correctly after any calls made. Instead, with ownership of the data in Frame, both can be handled in a single place. Handling both in fn run() is more natural and leaves less room for runtime API bugs.

The returned output is reset each time before running any executable in a nested stack.
This change should not incur any overhead to the overall memory usage as only the returned data from the last executed frame will be kept around at any time.

[#6261]: Add missing events to identity pallet

Extrinsics from pallet_identity that were missing an event emission on success now emit one.

[#5701]: [pallet-revive] uapi: allow create1 equivalent calls

The salt argument should be optional to allow create1 equivalent calls.

[#5555]: Make salt optional

Remove address_len and salt_len from uapi as both are now fixed size

[#5630]: Introduce and Implement the `VestedTransfer` Trait

This PR introduces a new trait VestedTransfer which is implemented by pallet_vesting. With this, other pallets can easily introduce vested transfers into their logic.

[#4257]: Rename `state_version` in `RuntimeVersion` to `system_version`.

This PR renames state_version in RuntimeVersion to system_version. system_version=2 signifies that extrinsic root derivation uses StateVersion::V1.

[#6812]: xcm-executor: take delivery fee from transferred assets if necessary

In asset transfers, as a last resort, XCM delivery fees are taken from transferred assets rather than failing the transfer.

[#6360]: [eth-rpc] proxy /health

make the eth-rpc proxy /health and /health/readiness from the proxied substrate chain see #4802

[#6147]: [pallet-revive] Ethereum JSON-RPC

Add a new Ethereum JSON-RPC server that can be used a substrate chain configured with pallet-revive

[#4982]: Add useful error logs in pallet-xcm

This PR adds error logs to assist in debugging pallet-xcm.
Additionally, it replaces the usage of log with tracing.

[#5311]: No-op Impl Polling Trait

Provide a NoOp implementation of the Polling trait for unit where the trait is defined and skiping benchmarks that necessitate it's definition.

[#6148]: Fix migrations for pallet-xcm

pallet-xcm stores some operational data that uses Versioned* XCM types. When we add a new XCM version (XV), we deprecate XV-2 and remove XV-3.
This PR extends the existing MigrateToLatestXcmVersion to include migration for the Queries, LockedFungibles, and RemoteLockedFungibles storage types.
Additionally, more checks were added to try_state for these types.

[#5675]: [pallet-revive] Add balance_of syscyall for fetching foreign balances

This adds an API method balance_of, corresponding to the BALANCE EVM opcode.

[#6291]: migrate pallet-remarks to v2 bench syntax

Part of:

#6202

[#6023]: Fix storage in pallet section

Fix compilation of pallet::storage in a pallet section: a local binding definition was not correctly referenced due to macro hygiene.

[#5861]: [pallet-revive] immutable data storage

This PR introduces the concept of immutable storage data, used for
Solidity immutable variables.

This is a minimal implementation. Immutable data is attached to a contract; to ContractInfo fixed in size, we only store the length there, and store the immutable data in a dedicated storage map instead. Which comes at the cost of requiring an storage read (costly) for contracts using this feature.

We discussed more optimal solutions not requiring any additional storage accesses internally, but they turned out to be non-trivial to implement. Another optimization benefiting multiple calls to the same contract in a single call stack would be to cache
the immutable data in Stack. However, this potential creates a DOS vulnerability (the attack vector is to call into as many contracts in a single stack as possible, where they all have maximum immutable data to fill the cache as efficiently as possible). So
this either has to be guaranteed to be a non-issue by limits, or, more likely, to have some logic to bound the cache. Eventually, we should think about introducing the concept of warm and cold storage reads (akin to EVM). Since immutable variables are commonly
used in contracts, this change is blocking our initial launch and we should only optimize it properly in follow-ups.

This PR also disables the set_code_hash API (which isn't usable for Solidity contracts without pre-compiles anyways). With immutable storage attached to contracts, we now want to run the constructor of the new code hash to collect the immutable data during set_code_hash. This will be implemented in a follow up PR.

[#5995]: Use frame umbrella crate in pallet-proxy and pallet-multisig

Extends the FRAME umbrella crate and uses it in pallet-proxy and pallet-multisig.
Migrates benchmarking from v1 to v2 for pallet-proxy and pallet-multisig.
Allows CI to pick the umbrella crate weights template to run benchmarks.

[#6299]: migrate pallet-recovery to benchmark V2 syntax

migrate pallet-recovery to benchmark V2 syntax

[#5872]: [omni-bencher] Make all runtimes work

Changes:

Add --exclude-pallets to exclude some pallets from runtimes where we dont have genesis presets yet
Make --genesis-builder-policy=none work with --runtime
CI: Run the frame-omni-bencher for all runtimes

[#5554]: Identity Decouple usernames from identities

This PR refactors pallet-identity to decouple usernames from identities. Usernames are now separated from identities in storage, allowing for correct deposit accounting and for authorities to put up their own deposit to create a username and remove usernames. Various storage maps had to be refactored and migrated to allow this to happen. The call to remove a dangling username is now replaced by the permissioned kill_username call.

[#6217]: Unify and harden UMP signal checks in check_core_index

Refactors and hardens the core index checks on the candidate commitments.
Also adds a utility for skipping the ump signals

[#5664]: Calling an address without associated code is a balance transfer

This makes pallet_revive behave like EVM where a balance transfer is just a call to a plain wallet.

[#3881]: Introduce a Generic Proving Trie

This PR introduces a Proving Trie object which can be used inside the runtime. This can allow for things like airdrops where a single hash is stored on chain representing the whole airdrop and individuals present a proof of their inclusion in the airdrop.

[#6365]: pallet-revive: Use `RUSTUP_TOOLCHAIN` if set

We were not passing through the RUSTUP_TOOLCHAIN variable to the build.rs script of our fixtures. This means that setting the toolchain like cargo +1.81 build had no effect on the fixture build. It would always fall back to the default toolchain.

[#5804]: Refactor get_account_id_from_seed / get_from_seed to one common place

get_account_id_from_seed / get_from_seed were copied all over the place. This PR removes unnecessary code duplication.
Keyring::iter() provides the same functionality and is used instead.

[#5886]: Bump some dependencies

This bumps ethbloom, ethereum-types, primitive-types and rlp to their latest version.

Fixes: #5870

[#5807]: [pallet-revive] last call return data API

This PR adds the EVM chain ID to Config as well as a corresponding runtime API so contracts can query it.

Related issue: paritytech/revive#44

[#5679]: Switch to new `CandidateReceipt` primitives

This change is just plumbing work and updates all crate interfaces to use the new primitives.
It doesn't alter any functionality and is required before implementing RFC103 on the node side.

[#6129]: Improved TrustedQueryAPI signatures.

Changed returned type of API methods from Result<bool, xcm_runtime_apis::trusted_query::Error> to a typed one:
type XcmTrustedQueryResult = Result<bool, xcm_runtime_apis::trusted_query::Error>

[#6473]: add `TransactionSource` to `TransactionExtension::validate`

Add a the source of the extrinsic as an argument in TransactionExtension::validate.
The transaction source can be useful for transactions that should only be valid if it comes from the node. For example from offchain worker.
To update the current code. The transaction source can simply be ignored: _source: TransactionSource

[#5838]: enable wasm builder diagnostics propagation

substrate-wasm-builder is used as a build dependency by crates that implement FRAME runtimes.
Errors that occur in these crates can not be detected by IDEs that use rust-analyzer as a language server because rust-analyzer needs the errors to be reported as diagnostic message in json format to be able to publish them to language server clients. This PR adds WASM_BUILD_CARGO_ARGS environment variable, which can hold a space separated list of args that will be parsed and passed to the cargo command that it is used for building against wasm target. It can be used for the stated initial case, but it is also flexible enough to allow passing other arguments or formatting the messages using another available type.

[#5682]: Introduces `VerifyExistenceProof` trait

Introduces VerifyExistenceProof trait for verifying proofs in the runtime.
An implementation of the trait for binary and 16 patricia merkle tree is provided.

[#5939]: [pallet-revive] Bump PolkaVM and add static code validation

Statically validate basic block sizes and instructions.

[#5839]: Remove internal workaround for compiler bug

[#5423]: Runtime support for candidate receipt v2 (RFC103)

Implementation of RFC103 in the relay chain runtime.
The runtime will accept and validate the new receipts only if the FeatureIndex::CandidateReceiptV2 feature bit is enabled.

[#5693]: Remove `sp_runtime::RuntimeString` and replace with `Cow<'static, str>` or `String` depending on use case

Deprecate RuntimeString, replace with String or Cow<'static, str> where appropriate.

For downstream projects the upgrade will primarily consist of following two changes:

#[sp_version::runtime_version]
pub const VERSION: RuntimeVersion = RuntimeVersion {
-	spec_name: create_runtime_str!("statemine"),
-	impl_name: create_runtime_str!("statemine"),
+	spec_name: alloc::borrow::Cow::Borrowed("statemine"),
+	impl_name: alloc::borrow::Cow::Borrowed("statemine"),

		fn dispatch_benchmark(
			config: frame_benchmarking::BenchmarkConfig
-		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, sp_runtime::RuntimeString> {
+		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, alloc::string::String> {

SCALE encoding/decoding remains the same as before, but serde encoding in runtime has changed from bytes to string (it was like this in std environment already).

[#6105]: [pallet-revive] implement tx origin API

Implement a syscall to retreive the transaction origin.

[#5684]: [pallet-revive]

Update xcm runtime api, and fix pallet-revive xcm tests

[#5420]: XCMv5 - Better fee mechanism

In XCMv5, there's a new instruction, PayFees, which is meant to be a replacement for BuyExecution.
This instruction takes only one parameter, the asset that you are willing to use for fee payment.
There's no parameter for limiting the weight, the amount of the asset you put in is the limit of how much you're willing to pay.
This instruction works much better with delivery fees.
BuyExecution will still be around to ensure backwards-compatibility, however, the benefits of the new instruction are a good incentive to switch.
The proposed workflow is to estimate fees using the XcmPaymentApi and DryRunApi, then to put those values in PayFees and watch your message go knowing you covered all the necessary fees.
You can add a little bit more just in case if you want.
RefundSurplus now gets back all of the assets that were destined for fee payment so you can deposit them somewhere.
BEWARE, make sure you're not sending any other message after you call RefundSurplus, if not, it will error.

[#6781]: Bridges - revert-back congestion mechanism

With permissionless lanes PR#4949, the congestion mechanism based on sending Transact(report_bridge_status(is_congested)) from pallet-xcm-bridge-hub to pallet-xcm-bridge-hub-router was replaced with a congestion mechanism that relied on monitoring XCMP queues. However, this approach could cause issues, such as suspending the entire XCMP queue instead of isolating the affected bridge. This PR reverts back to using report_bridge_status as before.

[#5954]: templates: make node compilation optional

Node compilation for minimal and parachain templates is made optional, not part of the templates default-members list. At the same time, we introduce OmniNode as alternative to run the templates.

[#5946]: [FRAME] fix: Do not emit `Issued { amount: 0 }` event

Filter out Issued events in pallet-balances module when its balance amount is zero.

[#6296]: Migrate pallet-glutton benchmark to v2

Update pallet-glutton to benchmarks v2.

[#6809]: XCMv5 - SetHints instruction

Implementation of fellowship RFC 107.
The new SetHints instruction is a repackaging of SetAssetClaimer that also allows future "hints" which alter the default behaviour of the executor.
The AllowTopLevelPaidExecutionFrom barrier allows this instruction between WithdrawAsset and BuyExecution/PayFees to configure things before the actual meat of the program.

[#5892]: Treasury: add migration to clean up unapproved deprecated proposals

It is no longer possible to create Proposals storage item in pallet-treasury due to migration from governance v1 model but there are some Proposals whose bonds are still on hold with no way to release them.
The purpose of this migration is to clear Proposals which are stuck and return bonds to the proposers.

[#5756]: Improve APIs for Tries in Runtime

This PR introduces a trait ProvingTrie which has all the function you need to use tries in the runtime.
This trait includes the ability to create, query, and prove data in a trie. Another trait ProofToHashes allows developers to express the computational complexity of proof verification using the proof data.

[#6027]: Remove pallet::getter from pallet-offences

This PR removes pallet::getter from pallet-offences from type Reports. It also adds a test to verify that retrieval of Reports still works with storage::getter.

[#6885]: Omni-node: Detect pending code in storage and send go ahead signal in dev-mode.

When using the polkadot-omni-node with manual seal (--dev-block-time), it is now possible to perform runtime upgrades. The node will detect the pending validation code and send a go-ahead signal to the parachain.

[#6205]: pallet-message-queue: Fix max message size calculation

The max size of a message should not depend on the weight left in a given execution context. Instead the max message size depends on the service weights configured for the pallet. A message that may does not fit into on_idle is not automatically overweight, because it may can be executed successfully in on_initialize or in another block in on_idle when there is more weight left.

[#3970]: Update Treasury to Support Block Number Provider

The goal of this PR is to have the treasury pallet work on a parachain which does not produce blocks on a regular schedule, thus can use the relay chain as a block provider. Because blocks are not produced regularly, we cannot make the assumption that block number increases monotonically, and thus have new logic to handle multiple spend periods passing between blocks. To migrate existing treasury implementations, simply add type BlockNumberProvider = System to have the same behavior as before.

[#6141]: Improve `CheckMetadataHash` transaction extension weight and logic

The compilation now panics if the optional compile-time environment variable RUNTIME_METADATA_HASH contains an invalid value.
The weight for the CheckMetadataHash transaction extension is more accurate as it is almost compile-time.

[#6212]: Added Trusted Query API calls for Westend and Rococo chains

Added is_trusted_reserve and is_trusted_teleporter API calls to relay chains.
Given an asset and a location, they return if the chain trusts that location as a reserve or teleporter for that asset respectively.
You can implement them on your runtime by simply calling a helper function on pallet-xcm.

	impl xcm_runtime_apis::trusted_query::TrustedQueryApi<Block> for Runtime {
  fn is_trusted_reserve(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_reserve(asset, location)
  }
  fn is_trusted_teleporter(asset: VersionedAsset, location: VersionedLocation) -> Result<bool, xcm_runtime_apis::trusted_query::Error> {
    PolkadotXcm::is_trusted_teleporter(asset, location)
  }
}

[#6470]: XCMv5 - Add ExecuteWithOrigin instruction

Added a new instruction to XCMv5, ExecuteWithOrigin, that allows you to specify an interior origin and a set of instructions that will be executed using that origin.
The origins you can choose are None to clear it during the execution of the inner instructions, or Some(InteriorLocation) to descend into an interior location.
These two options mimic the behaviour of ClearOrigin and DescendOrigin respectively.
Crucially, this instruction goes back to the previous origin once the execution of those inner instructions end.
This allows use-cases like a parent location paying fees with one interior location, fetching funds with another, claiming assets on behalf of many different ones, etc.

[#5322]: Elastic scaling - introduce new candidate receipt primitive

[#5623]: Generic slashing side-effects

What?
Make it possible for other pallets to implement their own logic when a slash on a balance occurs.

How?
First we abstract the done_slash function of holds::Balanced to it's own trait that any pallet can implement.
Then we add a config type in pallet-balances that accepts a callback tuple of all the pallets that implement this trait.
Finally implement done_slash for pallet-balances such that it calls the config type.
Integration
The default implementation of done_slash is still an empty function, and the new config type of pallet-balances can be set to an empty tuple, so nothing changes by default.

[#6384]: Relax requirements on `assign_core`.

Relax requirements for assign_core so that it accepts updates for the last scheduled entry.
This will allow the coretime chain to split up assignments into multiple messages, which allows for interlacing down to single block granularity.

Fixes: #6102

Changelog for `Node Operator`

ℹ️ These changes are relevant to: Those who don't write any code and only run code.

[#6760]: chainHead: Always report discarded items for storage operations

This PR ensures that substrate always reports discarded items as zero.
This is needed to align with the rpc-v2 spec

[#5515]: Add retry logic in relay chain rpc interface

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

Types of extrinsic are now therefore

Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
- Bare transactions (deprecated) - Gossiped, validated with ValidateUnsigned (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
- Inherents - Not gossiped, validated with ProvideInherent.
Extended (Extra data) - Gossiped, validated via TransactionExtension.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).

Notable information on TransactionExtension and the differences from SignedExtension

AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate.
pre_dispatch is renamed to prepare.
validate runs transaction validation logic both off-chain and on-chain, and is non-mutating.
prepare runs on-chain pre-execution logic using information extracted during validation and is mutating.
validate and prepare are now passed an Origin rather than an AccountId. If the extension logic presumes an AccountId, consider using the trait function
AsSystemOriginSigner::as_system_origin_signer.
A signature on the underlying transaction may validly not be present.
The origin may be altered during validation.
Validation functionality present in validate should not be repeated in prepare.
Useful information obtained during validate should now be passed in to prepare using the new user-specifiable type Val.
Unsigned logic should be temporarily migrated from the old *_unsigned functions into the regular versions of the new functions where the Origin is None, until the deprecation of ValidateUnsigned in phase 2 of Extrinsic Horizon.
The Call type defining the runtime call is now a type parameter.
Extensions now track the weight they consume during validation, preparation and post-dispatch through the TransactionExtensionBase::weight function.
TestXt was removed and its usage in tests was replaced with UncheckedExtrinsic instances.

[#6058]: backpressure `chainhead_v1_follow`

The RPC endpoint chainHead_v1_follow now relies on backpressure to determine whether or not the subscription should be closed instead of continuing to send more events to a consumer which can't keep up.
This should significantly improve memory consumption as substrate will be keeping less messages in memory.

[#5606]: Fix PVF precompilation for Kusama

Tweaks the PVF precompilation on Kusama to allow prepare PVFs when the node is an authority but not a validator.

[#6603]: Always provide main protocol name in litep2p responses

[#5741]: make RPC endpoint `chainHead_v1_storage` faster

The RPC endpoint chainHead_v1_storage now relies solely on backpressure to determine how quickly to serve back values instead of handing back a fixed number of entries and then expecting the client to ask for more. This should improve the throughput for bigger storage queries significantly.

Benchmarks using subxt on localhost:

Iterate over 10 accounts on westend-dev -> ~2-3x faster
Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
Fetch 1024 descendant values -> ~500x faster

[#5616]: PVF: drop backing jobs if it is too late

Introduces the removal of backing jobs that have been back pressured for longer than allowedAncestryLen, as these candidates are no longer viable.

[#5998]: Fix memory leak in litep2p public addresses

[#6527]: Update litep2p network backend to version 0.8.1

Release 0.8.1 of litep2p includes critical fixes to further enhance the stability and performance of the litep2p network backend.

[#5676]: [ci] Update CI image with rust 1.81.0 and 2024-09-11

cc https://github.com/paritytech/ci_cd/issues/1035

close https://github.com/paritytech/ci_cd/issues/1023

[#5859]: Add number of live peers available for requests

This PR adds a new metric for the number of live peers available for beefy requests.
The metric is exposed under the name substrate_beefy_on_demand_live_peers.

[#4639]: Added the fork-aware transaction pool implementation

New command line option was added, allowing to select implementation of transaction pool:
- --pool-type=fork-aware - new fork aware transaction pool,
- --pool-type=single-state - old transaction pool implementation which is still default,

[#4889]: Add CLI options for parachain chain specifications + fix bug for swallowing custom fields

Parachain ID and relay chain can be specified via the CLI arguments for when creating a chain spec.
A bug that also swallowed custom fields outside of the default config has also been fixed.

[#6353]: Update litep2p network backend to version 0.8.0

Release 0.8.0 of litep2p includes several improvements and memory leak fixes enhancing the stability and performance of the litep2p network backend.

[#6454]: rpc server: fix ipv6 host filter for localhost

This PR fixes that ipv6 connections to localhost was faulty rejected by the host filter because only [::1] was allowed

[#6380]: Do not propagate external addr with different peerIDs

External addresses that belong to a different peerID are no longer propagated to the higher layers of the networking backends.

[#6298]: Populate authority DHT records with public listen addresses

This PR populates the authority DHT records with public listen addresses if any.
The change effectively ensures that addresses are added to the DHT record in the following order:

Public addresses provided by CLI --public-addresses
Maximum of 4 public (global) listen addresses (if any)
Any external addresses discovered from the network (ie from /identify protocol)

While at it, this PR adds the following constraints on the number of addresses:

Total number of addresses cached is bounded at 16 (increased from 10).
A maximum number of 32 addresses are published to DHT records (previously unbounded).
A maximum of 4 global listen addresses are utilized.

This PR replaces the following warning:
WARNING: No public address specified, validator node may not be reachable. with a more descriptive one originated from the authority-discovery mechanism itself: No public addresses configured and no global listen addresses found.

[#6016]: Litep2p network backend do not disconnect all peers on SetReservedPeers command

[#6406]: make prospective-parachains debug logs less spammy

Demote some of the frequent prospective-parachains debug logs to trace level and prefer printing aggregate debug logs.

[#5572]: added RPC metrics for the collator

[#6860]: Update litep2p network backend to v0.8.4

This PR updates the Litep2p network backend to version 0.8.4

[#5609]: Update litep2p network backend to v0.7.0

The Identify protocol no longer includes public addresses in its configuration.
Instead, we rely on the litep2p.public_addresses interface to propagate external addresses of the node.

Litep2p uses hickory DNS resolver (formerly known as trust DNS).
Similarly to the trust DNS, the hickory logs are silenced.

[#5875]: Remove jaeger from polkadot

Jaeger was remove from the codebase because it was not used by anyone and it did not help with the debugging.

Changelog for `Runtime User`

ℹ️ These changes are relevant to: Anyone using the runtime. This can be a token holder or a dev writing a front end for a chain.

[#5961]: Bounties Pallet: add `approve_bounty_with_curator` call

[#4851]: Add support for deprecation metadata in `RuntimeMetadataIr` entries.

Changes introduced are listed below.
Adds DeprecationStatusIR enum to sp_metadata_ir.

Is a deprecation info for simple items.
Adds DeprecationInfoIR enum to sp_metadata_ir.
It is a deprecation info for an enums/errors/calls. Contains DeprecationStatusIR.
Also denotes full/partial deprecation of the type or its variants/calls.
Adds deprecation_info field to
RuntimeApiMetadataIR
RuntimeApiMethodMetadataIR
StorageEntryMetadataIR
PalletConstantMetadataIR
PalletCallMetadataIR
PalletMetadataIR
PalletEventMetadataIR
PalletErrorMetadataIR
Examples of the deprecation info produced can be seen inside
Tests for frame-support
hackmd link https://hackmd.io/@Zett98/Bys0YgbcR

[#3151]: Dynamic deposit based on number of proposals

Introduce a dynamic proposal deposit mechanism influenced by the total number of active proposals, with the option to set the deposit to none.

Two new calls are introduced:

kill(origin, proposal_hash): the cancellation of a proposal, accompanied by the burning of the associated cost/consideration ticket.
release_proposal_cost(origin, proposal_hash): the release of the cost for a non-active proposal.

New config parameters:

DisapproveOrigin: origin from which a proposal in any status may be disapproved without associated cost for a proposer;
KillOrigin: Origin from which any malicious proposal may be killed with associated cost for a proposer;
Consideration: mechanism to assess the necessity of some cost for publishing and storing a proposal. Set to unit type to have not submission cost;

Additionally change:

benchmarks have been upgraded to benchmarks::v2 for collective pallet;
ensure_successful function added to the Consideration under runtime-benchmarks feature.

[#4826]: XCMv5

Added XCMv5.

This PR brings a new XCM version.
It's an amalgamation of multiple individual PRs:

XCMv5 reduces the potential for bugs by:

Removing the need to specify weight in Transact.
Handling fees in a better way with PayFees instead of BuyExecution.
Improves asset claiming with SetAssetClaimer.

It also allows some new use-cases like:

Sending both teleported and reserve asset transferred assets in the same cross-chain transfer.
Preserving the origin when doing cross-chain transfers. Allowing the use of Transact in the same message as a cross-chain transfer.

[#5765]: Added foreign locations to local accounts converter to all the parachains.

Now any user account can have a sovereign account on another chain controlled by the original account.

[#6337]: Don't expose metadata for Runtime APIs that haven't been implemented

Prior to this PR, the metadata for runtime APIs would contain all methods for the latest version of each API, regardless of which version a runtime implements. This PR fixes that, so that the runtime API metadata reflects what is actually implemented.

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

Types of extrinsic are now therefore

Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned")
- Bare transactions (deprecated) - Gossiped, validated with ValidateUnsigned (deprecated) and the _bare_compat bits of TransactionExtension (deprecated).
- Inherents - Not gossiped, validated with ProvideInherent.
Extended (Extra data) - Gossiped, validated via TransactionExtension.
- Signed transactions (with a hardcoded signature).
- General transactions (without a hardcoded signature).

Notable information on TransactionExtension and the differences from SignedExtension

AdditionalSigned/additional_signed is renamed to Implicit/implicit. It is encoded for the entire transaction and passed in to each extension as a new argument to validate.
pre_dispatch is renamed to prepare.
validate runs transaction validation logic both off-chain and on-chain, and is non-mutating.
prepare runs on-chain pre-execution logic using information extracted during validation and is mutating.
validate and prepare are now passed an Origin rather than an AccountId. If the extension logic presumes an AccountId, consider using the trait function
AsSystemOriginSigner::as_system_origin_signer.
A signature on the underlying transaction may validly not be present.
The origin may be altered during validation.
Validation functionality present in validate should not be repeated in prepare.
Useful information obtained during validate should now be passed in to prepare using the new user-specifiable type Val.
Unsigned logic should be temporarily migrated from the old *_unsigned functions into the regular versions of the new functions where the Origin is None, until the deprecation of ValidateUnsigned in phase 2 of Extrinsic Horizon.
The Call type defining the runtime call is now a type parameter.
Extensions now track the weight they consume during validation, preparation and post-dispatch through the TransactionExtensionBase::weight function.
TestXt was removed and its usage in tests was replaced with UncheckedExtrinsic
instances.

[#6061]: Remove check-migrations for rococo chain

This PR adds the missing cumulus_pallet_xcmp_queue v5 migration to the coretime-westend runtime.

[#5971]: XCMv5 InitiateTransfer can preserve original origin across chains.

The new InitiateTransfer instruction can preserve the original origin across chains by setting preserve_origin: true in the instruction itself.
When it's set to true, it will append after the inner XCM, an AliasOrigin instruction instead of the usual ClearOrigin.
This instruction will try to alias to the original origin, thus preserving it.
This only works if the chain receiving the transfer supports the aliasing operation.
If not, preserve_origin: false works as before and will never fail because of this.

[#6373]: `chain-spec-builder`: info about patch/full files added

There was no good example of what is patch and full genesis config file. Some explanation and example were added to the chain-spec-builder doc.

[#6357]: New runtime api that returns the associated pool accounts with a nomination pool.

Each nomination pool has two associated pot accounts: the bonded account, where funds are pooled for staking, and the reward account. This update introduces a runtime api that clients can query to retrieve these accounts.

[#6039]: Added Trusted Query API calls.

There's a new runtime API to check if a chain trust a Location as a reserve or teleporter for a given Asset.
It's implemented in all the relays and system parachains in Westend and Rococo.

[#5198]: MQ processor should be transactional

Enforce transactional processing on pallet Message Queue Processor.

Storage changes that were done while processing a message will now be rolled back when the processing returns an error. Ok(false) will not revert, only Err(_).

[#6228]: Transact without having to specify weight

In XCMv5, it's no longer required to pass in the expected weight when using Transact.
This was made to remove a whole class of bugs where the weight specified was not enough.

[#5390]: Remove NetworkIds for testnets Rococo and Westend

Implemetation of polkadot-fellows/RFCs#108, in the version 5 of XCM, Remove Westend and Rococo from the included NetworkIds to improve the stability of the language.

NetworkId::Rococo and NetworkId::Westend can just use NetworkId::ByGenesis by importing their genesis block hash

[#5857]: Beefy equivocation: check all the MMR roots

[#5435]: Support registering assets on Asset Hubs over bridge

Allows one Asset Hub on one side, to register assets on the other Asset Hub over the bridge.
Rococo <> Ethereum test bridge will be dropped in favor of Westend <> Ethereum test bridge.
This PR also changes emulated tests to simulate double bridging from Ethereum<>Westend<>Rococo.

[#4257]: Rename `state_version` in `RuntimeVersion` to `system_version`.

RuntimeVersion's state_version is renamed to system_version. Applications using that type and its field must update their code to reflect the changes. For easier migration serde serialization produces both new systemVersion and old stateVersion fields and deserialization supports stateVersion as an alias as too.

[#5984]: Add page information to staking::PayoutStarted event

Adds page index that is claimed, and optional next page that can be claimed. If next is none, then the page is the last one.

[#5554]: Identity Decouple usernames from identities

[#6022]: [Coretime chain] Add high assignment count mitigation to testnets

We can handle a maximum of 28 assignments inside one XCM, while it's possible to have 80 (if a region is interlaced 79 times). This can be chunked on the coretime chain side but currently the relay does not support this. This PR truncates the additional assignments on Rococo and Westend to mitigate this until the relay is fixed. The first 27 assignments are taken, the final 28th is used to pad with idle to complete the mask. Any other assignments are dropped.

[#6080]: Assets in pool with native can be used in query_weight_to_asset_fee in Asset Hubs

query_weight_to_asset_fee now works with assets in a pool with the native asset in both Westend and Rococo asset hubs.
This means all the information you get from query_acceptable_payment_assets can be used directly in query_weight_to_asset_fee to get the correct fees that need to be paid.

[#5420]: XCMv5 - Better fee mechanism

[#5946]: [FRAME] fix: Do not emit `Issued { amount: 0 }` event

Filter out Issued events in pallet-balances module when its balance amount is zero.

[#5687]: Westend/Rococo Asset Hub: auto incremented asset id for trust backed assets

Setup auto incremented asset id to 50_000_000 for trust backed assets.

Migration

This change does not break the API but introduces a new constraint. It implements an auto-incremented ID strategy for Trust-Backed Assets (50 pallet instance indexes on both networks), starting at ID 50,000,000. Each new asset must be created with an ID that is one
greater than the last asset created. The next ID can be fetched from the NextAssetId storage item of the assets pallet. An empty NextAssetId storage item indicates no constraint on the next asset ID and can serve as a feature flag for this release.

[#6212]: Added Trusted Query API calls for Westend and Rococo chains

There's a new runtime API to check if a chain trust a Location as a reserve or teleporter for a given Asset.
It's implemented in all the relays and system parachains in Westend and Rococo.

[#6470]: XCMv5 - Add ExecuteWithOrigin instruction

[#5999]: Westend: Constant yearly emission

Integrating the new inflation approach from polkadot-fellows/runtimes#471 into Westend first to check that it is working.

Rust compiler versions

This release was built and tested against the following versions of rustc.
Other versions may work.

Rust Stable: "1.81.0"

Runtimes

The information about the runtimes included in this release can be found below.
The runtimes have been built using srtool v0.17.0 and rustc 1.81.0 (eeb90cda1 2024-09-04).

Westend

🏋️ Runtime size:             2.037 MB (2,135,690 bytes)
🗜 Compressed:               Yes, 73.61%
✨ Reserved meta:            OK - [6D, 65, 74, 61]
🎁 Metadata version:         V14
🔥 Core version:             westend-1017001 (parity-westend-0.tx27.au2)
🗳️ system.setCode hash:      0x66982e5b4340ac74a19a5a51ae7774ac05bcd5ef502f6b19a5f18d28afcce211
🗳️ authorizeUpgrade hash:    0xfcb5c327345609ecaa45bd679959f6d15a3c621c5ba6822af31c373c7ab5140c
🗳️ Blake2-256 hash:          0xfd9e364b5ee6fc709af5a746557ae9e5f8d5357c7cfcc50058f3b3bbd40303d5
📦 IPFS:                     QmY1mCdSL2CFFGALgPGbkzQxz6qH7GSqVJp31ye1EZc4RQ

Westend Assethub

🏋️ Runtime Size:          1.60 MB (1674969 bytes)
🔥 Core Version:          westmint-1017003 (westmint-0.tx16.au1)
🗜 Compressed:            Yes: 76.65%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0xd522fe21a324cf368516f02ba9f9d95c90881a0c44559a8ddb44cc13c7a05732
🗳️ authorizeUpgrade hash: 0x2150f7b02db7513f50305a16bc2a6723201f89c9c4d99967979a6da7b9848e1a
🗳️ Blake2-256 hash:       0x29bac726f658b40db916045e89ba2a5c6f404cfb266d4a24539740290bcfcb79
📦 IPFS:                  QmXkckKSkkxxdzdPemTngGNNThfhUCCXU7rZJhyupqyDws

Westend Bridgehub

🏋️ Runtime Size:          1.38 MB (1450346 bytes)
🔥 Core Version:          bridge-hub-westend-1017001 (bridge-hub-westend-0.tx6.au1)
🗜 Compressed:            Yes: 73.72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x52715185f7f28290fd05172e5b0f016ea0dac4808265323e6a23ba6368d80b0c
🗳️ authorizeUpgrade hash: 0xba8101845222ce0715fd4c063c9972e85b173c48c317a025bc1b969e1cbfe03d
🗳️ Blake2-256 hash:       0xcdc83f4c85dad25c3dfbfbce1ea57068b68105d4f95b29a677287bc527183aaf
📦 IPFS:                  QmfJn3gHYWdKsdnRUtZTB3MdFgSN7ihH1wQS4duNDStWVF

Westend Collectives

🏋️ Runtime Size:          1.21 MB (1267199 bytes)
🔥 Core Version:          collectives-westend-1017001 (collectives-westend-0.tx6.au1)
🗜 Compressed:            Yes: 75.58%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0xb064f17227559bf5fb5010c8a8ade578458da1ff1a228d66cf315fc10108b7bf
🗳️ authorizeUpgrade hash: 0x4bd35a76c3e53e93025ed4a3269deff9e01a60b1936414445f78b93f2f90183d
🗳️ Blake2-256 hash:       0xb96fee4c33e0f9a5bcb10700a971b542abe94a9f9e9e4a7e25ba5b8442c51313
📦 IPFS:                  QmVeSPx4KDMkt1yxfLKXozYaNDw3Qi5zZnJ5NmW6Dkh95w

Westend Coretime

🏋️ Runtime Size:          1.03 MB (1080704 bytes)
🔥 Core Version:          coretime-westend-1017001 (coretime-westend-0.tx2.au1)
🗜 Compressed:            Yes: 74.72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x9fad76f381887a24ef60bc7097c7ae109a5f31c3e5509132d274c40e61304e93
🗳️ authorizeUpgrade hash: 0xc5e10106007862b04a990da6f94288c46e9ccbeaa73e118365b93e0f35320df9
🗳️ Blake2-256 hash:       0x76973fbaf5b3c5da03b2f517b7385dafd51596f3d351e243bbbf5bf06443fdeb
📦 IPFS:                  QmT97ek69zN6nBBNUi1YD7o37L3RTEe8q1Bf3MipJaiKU6

Westend Glutton

🏋️ Runtime Size:          586.78 kB (600867 bytes)
🔥 Core Version:          glutton-westend-1017001 (glutton-westend-0.tx1.au1)
🗜 Compressed:            Yes: 72%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x9a7fc73bd2e6858eeead2f237e5bde6f3e75e942fa1465c54d7dbf1204e7145b
🗳️ authorizeUpgrade hash: 0x3a974c855ecdd0068e0a3c7b0e3f80a7b1488d7051af67b2c6c76334557a812d
🗳️ Blake2-256 hash:       0x60849ff400d4d0a7e8dd47fff1350b2799e97abb3d64957f0e8f5a838ca5a457
📦 IPFS:                  QmeAst2t3foRqmuwX5GGMkxErRet56eeczvK3bvkgdYPdy

Westend People

🏋️ Runtime Size:          1.03 MB (1077104 bytes)
🔥 Core Version:          people-westend-1017001 (people-westend-0.tx2.au1)
🗜 Compressed:            Yes: 74.78%
🎁 Metadata version:      V14
🗳️ system.setCode hash:   0x851eac6520032e3d957ecdff6b79989045117a0c59684243e4127a97ba203e3a
🗳️ authorizeUpgrade hash: 0x8bed1b714ce09aaa06f9948f0f893d5cccc97d838d8420fc3c0dc657c3242fa3
🗳️ Blake2-256 hash:       0x0352e12aece34ea64134a4cf8caa165961c8600aee2d5e69f2e511467a1e8009
📦 IPFS:                  QmSvKyCUCBsgVtXKXQWcsZfvqnenXCYBcFwPLZ91hPc8xf

Docker images

The docker images for the polkadot node binary and the polkadot-parachain binary can be found at Docker hub (will be available a few minutes after the release has been published):

You may also pull it with:

docker pull parity/polkadot:stable2412

docker pull parity/polkadot-parachain:stable2412

Polkadot stable2412

Changelog

Changelog for Node Dev

[#5883]: statement-distribution RFC103 implementation

[#6864]: Fix approval-voting canonicalize off by one

[#5919]: substrate-offchain: upgrade hyper to v1

[#5908]: collation-generation: use v2 receipts

[#6729]: Fix order of resending messages after restart

[#5372]: elastic scaling: add core selector to cumulus

[#5768]: export NodeHealthProxyLayer

[#5911]: Removed the possibility to start a shell parachain node

[#6760]: chainHead: Always report discarded items for storage operations

[#5515]: Add retry logic in relay chain rpc interface

[#5540]: Avoid unnecessary block gap updates

[#5635]: Fix edge case where state sync is not triggered

[#5469]: Syncing strategy refactoring

[#5774]: Avoid unnecessary state reset of allowed_requests when no block requests are sent

[#5038]: Plumb RPC listener up to caller

[#3685]: FRAME Reintroduce TransactionExtension as a replacement for SignedExtension

[#4837]: Add PVF execution priority

[#5521]: Allow to call arbitrary runtime apis using RelayChainInterface

[#5686]: sync: Remove checking of the extrinsics root

[#5601]: Introduce RpcParams in sc-cli

[#4849]: Introduce approval-voting-parallel subsystem

[#5830]: Remove jaeger from approval-voting and approval-distribution

[#6603]: Always provide main protocol name in litep2p responses

[#5915]: Omni-Node renamings

[#4974]: Remove libp2p dependency from sc-network-sync

[#5461]: runtime: remove ttl

[#6676]: Expose the unstable metadata v16

[#6218]: Enable approval-voting-parallel by default on kusama

[#6268]: Bump a timeout in zombienet coretime smoke test

[#5616]: PVF: drop backing jobs if it is too late

[#6156]: Use bool::then instead of then_some with function calls

[#5998]: Fix memory leak in litep2p public addresses

[#5824]: Bump parachains runtime API to v11

[#6527]: Update litep2p network backend to version 0.8.1

[#6646]: OmniNode --dev flag starts node with manual seal

[#5676]: [ci] Update CI image with rust 1.81.0 and 2024-09-11

[#5859]: Add number of live peers available for requests

[#6015]: Rename QueueEvent::StartWork

[#5732]: Expose the unstable metadata v16

[#4639]: Added the fork-aware transaction pool implementation

[#5924]: Bump PoV request timeout

[#5847]: candidate-validation: RFC103 implementation

[#6677]: chore: Update litep2p to v0.8.2

[#5592]: Introduce BlockGap

[#6353]: Update litep2p network backend to version 0.8.0

[#6690]: Fix Possible bug, Vote import failed after aggression is enabled

[#6652]: rpc server: re-use server builder per rpc interface

[#5856]: Extend state tracking of chainHead to capture notification gaps

[#6742]: Update litep2p backend to v0.8.3

[#6380]: Do not propagate external addr with different peerIDs

[#6217]: Unify and harden UMP signal checks in check_core_index

[#5880]: Fix prospective parachains test to use shuffled candidate list

[#5997]: Implement archive_unstable_storageDiff method

[#6298]: Populate authority DHT records with public listen addresses

[#5737]: Make syncing service an argument of build_network

[#5716]: Replace lazy_static with LazyLock

[#5679]: Switch to new CandidateReceipt primitives

[#5789]: Prevents EthereumBlobExporter from consuming parameters when returning NotApplicable

[#5839]: Remove internal workaround for compiler bug

[#5423]: Runtime support for candidate receipt v2 (RFC103)

[#6588]: rpc server: fix subscription id_provider being reset to default one

[#6016]: Litep2p network backend do not disconnect all peers on SetReservedPeers command

[#6382]: gensis-config: patching default RuntimeGenesisConfig fixed

[#5787]: Move bitfield_distribution to blocking task pool and set capacity to 8192

[#5762]: Fast return for invalid request of node health

[#5954]: templates: make node compilation optional

[#6406]: make prospective-parachains debug logs less spammy

[#5572]: added RPC metrics for the collator

[#6696]: Make approval-distribution aggression a bit more robust and less spammy

[#6860]: Update litep2p network backend to v0.8.4

[#6863]: Update merkleized-metadata to 0.2.0

[#5811]: Improve import_notification_stream documentation

[#5343]: Allow to disable gap creation during block import

[#5322]: Elastic scaling - introduce new candidate receipt primitive

[#5609]: Update litep2p network backend to v0.7.0

[#6104]: LocalTransactionPool implemented for fork aware transaction pool

[#5666]: Make syncing strategy an argument of the syncing engine

Changelog for `Node Dev`

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

[#5601]: Introduce `RpcParams` in sc-cli

[#5847]: `candidate-validation`: RFC103 implementation

[#5592]: Introduce `BlockGap`

[#5737]: Make syncing service an argument of `build_network`

[#5716]: Replace `lazy_static` with `LazyLock`

[#5679]: Switch to new `CandidateReceipt` primitives

[#6382]: gensis-config: patching default `RuntimeGenesisConfig` fixed

[#5811]: Improve `import_notification_stream` documentation

Changelog for `Runtime Dev`

[#5961]: Bounties Pallet: add `approve_bounty_with_curator` call

[#4851]: Add support for deprecation metadata in `RuntimeMetadataIr` entries.

[#6032]: Fix `feeless_if` in pallet section

[#6496]: Backport #6418 to `stable2412`

[#6645]: xcm: fix local/remote exports when inner routers return `NotApplicable`

[#6025]: Refactor staking pallet benchmarks to `v2`

[#6073]: Refactor `pallet-grandpa` benchmarks to `v2`

[#6169]: [Deprecation] deprecate treasury `spend_local` call and related items

For users who were using only `spend_local` before

For users who were already using `spend` with all other assets, except the native asset

[#3685]: FRAME Reintroduce `TransactionExtension` as a replacement for `SignedExtension`

[#5745]: Implement `try_append` for `StorageNMap`

[#4012]: `impl_runtime_apis!`: replace the use of `Self` with `Runtime`