XcmDryRunApi - Dry-running extrinsics to get their XCM effects (paritytech#3872)

# Context

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](paritytech#3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

# Main addition

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

# How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

# Dear reviewer

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

## Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Author: 2 people

Cargo.lock

@@ -37,8 +37,9 @@ use codec::Encode;
 use frame_support::traits::Get;
 use sp_core::H256;
 use sp_runtime::{FixedPointNumber, FixedU128, Saturating};
+use sp_std::vec::Vec;
 use xcm::prelude::*;
-use xcm_builder::{ExporterFor, SovereignPaidRemoteExporter};
+use xcm_builder::{ExporterFor, InspectMessageQueues, SovereignPaidRemoteExporter};
 
 pub use pallet::*;
 pub use weights::WeightInfo;
@@ -95,7 +96,7 @@ pub mod pallet {
 		/// Origin of the sibling bridge hub that is allowed to report bridge status.
 		type BridgeHubOrigin: EnsureOrigin<Self::RuntimeOrigin>;
 		/// Actual message sender (`HRMP` or `DMP`) to the sibling bridge hub location.
-		type ToBridgeHubSender: SendXcm;
+		type ToBridgeHubSender: SendXcm + InspectMessageQueues;
 		/// Underlying channel with the sibling bridge hub. It must match the channel, used
 		/// by the `Self::ToBridgeHubSender`.
 		type WithBridgeHubChannel: XcmChannelStatusProvider;
@@ -396,6 +397,12 @@ impl<T: Config<I>, I: 'static> SendXcm for Pallet<T, I> {
 	}
 }
 
+impl<T: Config<I>, I: 'static> InspectMessageQueues for Pallet<T, I> {
+	fn get_messages() -> Vec<(VersionedLocation, Vec<VersionedXcm<()>>)> {
+		ViaBridgeHubExporter::<T, I>::get_messages()
+	}
+}
+
 #[cfg(test)]
 mod tests {
 	use super::*;
@@ -635,4 +642,36 @@ mod tests {
 			);
 		});
 	}
+
+	#[test]
+	fn get_messages_works() {
+		run_test(|| {
+			assert_ok!(send_xcm::<XcmBridgeHubRouter>(
+				(Parent, Parent, GlobalConsensus(BridgedNetworkId::get()), Parachain(1000)).into(),
+				vec![ClearOrigin].into()
+			));
+			assert_eq!(
+				XcmBridgeHubRouter::get_messages(),
+				vec![(
+					VersionedLocation::V4((Parent, Parachain(1002)).into()),
+					vec![VersionedXcm::V4(
+						Xcm::builder()
+							.withdraw_asset((Parent, 1_002_000))
+							.buy_execution((Parent, 1_002_000), Unlimited)
+							.set_appendix(
+								Xcm::builder_unsafe()
+									.deposit_asset(AllCounted(1), (Parent, Parachain(1000)))
+									.build()
+							)
+							.export_message(
+								Kusama,
+								Parachain(1000),
+								Xcm::builder_unsafe().clear_origin().build()
+							)
+							.build()
+					)],
+				),],
+			);
+		});
+	}
 }

bridges/modules/xcm-bridge-hub-router/src/lib.rs

@@ -19,14 +19,16 @@
 use crate as pallet_xcm_bridge_hub_router;
 
 use bp_xcm_bridge_hub_router::XcmChannelStatusProvider;
+use codec::Encode;
 use frame_support::{
 	construct_runtime, derive_impl, parameter_types,
 	traits::{Contains, Equals},
 };
 use frame_system::EnsureRoot;
 use sp_runtime::{traits::ConstU128, BuildStorage};
+use sp_std::cell::RefCell;
 use xcm::prelude::*;
-use xcm_builder::{NetworkExportTable, NetworkExportTableItem};
+use xcm_builder::{InspectMessageQueues, NetworkExportTable, NetworkExportTableItem};
 
 pub type AccountId = u64;
 type Block = frame_system::mocking::MockBlock<TestRuntime>;
@@ -102,23 +104,46 @@ pub struct TestToBridgeHubSender;
 
 impl TestToBridgeHubSender {
 	pub fn is_message_sent() -> bool {
-		frame_support::storage::unhashed::get_or_default(b"TestToBridgeHubSender.Sent")
+		!Self::get_messages().is_empty()
 	}
 }
 
+thread_local! {
+	pub static SENT_XCM: RefCell<Vec<(Location, Xcm<()>)>> = RefCell::new(Vec::new());
+}
+
 impl SendXcm for TestToBridgeHubSender {
-	type Ticket = ();
+	type Ticket = (Location, Xcm<()>);
 
 	fn validate(
-		_destination: &mut Option<Location>,
-		_message: &mut Option<Xcm<()>>,
+		destination: &mut Option<Location>,
+		message: &mut Option<Xcm<()>>,
 	) -> SendResult<Self::Ticket> {
-		Ok(((), (BridgeFeeAsset::get(), HRMP_FEE).into()))
+		let pair = (destination.take().unwrap(), message.take().unwrap());
+		Ok((pair, (BridgeFeeAsset::get(), HRMP_FEE).into()))
 	}
 
-	fn deliver(_ticket: Self::Ticket) -> Result<XcmHash, SendError> {
-		frame_support::storage::unhashed::put(b"TestToBridgeHubSender.Sent", &true);
-		Ok([0u8; 32])
+	fn deliver(pair: Self::Ticket) -> Result<XcmHash, SendError> {
+		let hash = fake_message_hash(&pair.1);
+		SENT_XCM.with(|q| q.borrow_mut().push(pair));
+		Ok(hash)
+	}
+}
+
+impl InspectMessageQueues for TestToBridgeHubSender {
+	fn get_messages() -> Vec<(VersionedLocation, Vec<VersionedXcm<()>>)> {
+		SENT_XCM.with(|q| {
+			(*q.borrow())
+				.clone()
+				.iter()
+				.map(|(location, message)| {
+					(
+						VersionedLocation::V4(location.clone()),
+						vec![VersionedXcm::V4(message.clone())],
+					)
+				})
+				.collect()
+		})
 	}
 }
 
@@ -146,3 +171,7 @@ pub fn new_test_ext() -> sp_io::TestExternalities {
 pub fn run_test<T>(test: impl FnOnce() -> T) -> T {
 	new_test_ext().execute_with(test)
 }
+
+pub(crate) fn fake_message_hash<T>(message: &Xcm<T>) -> XcmHash {
+	message.using_encoded(sp_io::hashing::blake2_256)
+}

bridges/modules/xcm-bridge-hub-router/src/mock.rs

@@ -38,6 +38,7 @@ polkadot-parachain-primitives = { path = "../../../polkadot/parachain", default-
 polkadot-runtime-parachains = { path = "../../../polkadot/runtime/parachains", default-features = false }
 polkadot-runtime-common = { path = "../../../polkadot/runtime/common", default-features = false, optional = true }
 xcm = { package = "staging-xcm", path = "../../../polkadot/xcm", default-features = false }
+xcm-builder = { package = "staging-xcm-builder", path = "../../../polkadot/xcm/xcm-builder", default-features = false }
 
 # Cumulus
 cumulus-pallet-parachain-system-proc-macro = { path = "proc-macro", default-features = false }
@@ -95,6 +96,7 @@ std = [
 	"sp-tracing/std",
 	"sp-trie/std",
 	"trie-db/std",
+	"xcm-builder/std",
 	"xcm/std",
 ]
 
@@ -109,6 +111,7 @@ runtime-benchmarks = [
 	"polkadot-runtime-common/runtime-benchmarks",
 	"polkadot-runtime-parachains/runtime-benchmarks",
 	"sp-runtime/runtime-benchmarks",
+	"xcm-builder/runtime-benchmarks",
 ]
 
 try-runtime = [

cumulus/pallets/parachain-system/Cargo.toml

@@ -55,7 +55,8 @@ use sp_runtime::{
 	BoundedSlice, FixedU128, RuntimeDebug, Saturating,
 };
 use sp_std::{cmp, collections::btree_map::BTreeMap, prelude::*};
-use xcm::latest::XcmHash;
+use xcm::{latest::XcmHash, VersionedLocation, VersionedXcm};
+use xcm_builder::InspectMessageQueues;
 
 mod benchmarking;
 pub mod migration;
@@ -1612,6 +1613,19 @@ impl<T: Config> UpwardMessageSender for Pallet<T> {
 	}
 }
 
+impl<T: Config> InspectMessageQueues for Pallet<T> {
+	fn get_messages() -> Vec<(VersionedLocation, Vec<VersionedXcm<()>>)> {
+		use xcm::prelude::*;
+
+		let messages: Vec<VersionedXcm<()>> = PendingUpwardMessages::<T>::get()
+			.iter()
+			.map(|encoded_message| VersionedXcm::<()>::decode(&mut &encoded_message[..]).unwrap())
+			.collect();
+
+		vec![(VersionedLocation::V4(Parent.into()), messages)]
+	}
+}
+
 #[cfg(feature = "runtime-benchmarks")]
 impl<T: Config> polkadot_runtime_common::xcm_sender::EnsureForParachain for Pallet<T> {
 	fn ensure(para_id: ParaId) {

cumulus/pallets/parachain-system/src/lib.rs

@@ -28,6 +28,7 @@ polkadot-runtime-common = { path = "../../../polkadot/runtime/common", default-f
 polkadot-runtime-parachains = { path = "../../../polkadot/runtime/parachains", default-features = false }
 xcm = { package = "staging-xcm", path = "../../../polkadot/xcm", default-features = false }
 xcm-executor = { package = "staging-xcm-executor", path = "../../../polkadot/xcm/xcm-executor", default-features = false }
+xcm-builder = { package = "staging-xcm-builder", path = "../../../polkadot/xcm/xcm-builder", default-features = false }
 
 # Cumulus
 cumulus-primitives-core = { path = "../../primitives/core", default-features = false }
@@ -46,9 +47,6 @@ sp-core = { path = "../../../substrate/primitives/core" }
 pallet-balances = { path = "../../../substrate/frame/balances" }
 frame-support = { path = "../../../substrate/frame/support", features = ["experimental"] }
 
-# Polkadot
-xcm-builder = { package = "staging-xcm-builder", path = "../../../polkadot/xcm/xcm-builder" }
-
 # Cumulus
 cumulus-pallet-parachain-system = { path = "../parachain-system", features = ["parameterized-consensus-hook"] }
 
@@ -71,6 +69,7 @@ std = [
 	"sp-io/std",
 	"sp-runtime/std",
 	"sp-std/std",
+	"xcm-builder/std",
 	"xcm-executor/std",
 	"xcm/std",
 ]

cumulus/pallets/xcmp-queue/Cargo.toml

@@ -70,7 +70,8 @@ use scale_info::TypeInfo;
 use sp_core::MAX_POSSIBLE_ALLOCATION;
 use sp_runtime::{FixedU128, RuntimeDebug, Saturating};
 use sp_std::prelude::*;
-use xcm::{latest::prelude::*, VersionedXcm, WrapVersion, MAX_XCM_DECODE_DEPTH};
+use xcm::{latest::prelude::*, VersionedLocation, VersionedXcm, WrapVersion, MAX_XCM_DECODE_DEPTH};
+use xcm_builder::InspectMessageQueues;
 use xcm_executor::traits::ConvertOrigin;
 
 pub use pallet::*;
@@ -947,6 +948,38 @@ impl<T: Config> SendXcm for Pallet<T> {
 	}
 }
 
+impl<T: Config> InspectMessageQueues for Pallet<T> {
+	fn get_messages() -> Vec<(VersionedLocation, Vec<VersionedXcm<()>>)> {
+		use xcm::prelude::*;
+
+		OutboundXcmpMessages::<T>::iter()
+			.map(|(para_id, _, messages)| {
+				let mut data = &messages[..];
+				let decoded_format =
+					XcmpMessageFormat::decode_with_depth_limit(MAX_XCM_DECODE_DEPTH, &mut data)
+						.unwrap();
+				if decoded_format != XcmpMessageFormat::ConcatenatedVersionedXcm {
+					panic!("Unexpected format.")
+				}
+				let mut decoded_messages = Vec::new();
+				while !data.is_empty() {
+					let decoded_message = VersionedXcm::<()>::decode_with_depth_limit(
+						MAX_XCM_DECODE_DEPTH,
+						&mut data,
+					)
+					.unwrap();
+					decoded_messages.push(decoded_message);
+				}
+
+				(
+					VersionedLocation::V4((Parent, Parachain(para_id.into())).into()),
+					decoded_messages,
+				)
+			})
+			.collect()
+	}
+}
+
 impl<T: Config> FeeTracker for Pallet<T> {
 	type Id = ParaId;
 

cumulus/pallets/xcmp-queue/src/lib.rs

@@ -178,6 +178,7 @@ impl xcm_executor::Config for XcmConfig {
 	type HrmpNewChannelOpenRequestHandler = ();
 	type HrmpChannelAcceptedHandler = ();
 	type HrmpChannelClosingHandler = ();
+	type XcmRecorder = ();
 }
 
 pub type XcmRouter = (