Metadata V15 breaking change: Features to include in V15

[lexnv]

We would like to include the Runtime API info in the metadata and create a V15 release.
The metadata has been stable for a while and would like to minimize the number
of breaking changes from the user's perspective.

Considering this is a breaking change for the metadata, what other features would we
like to include in the metadata V15 before making a release?

Metadata with Runtime API

To help client-side applications make runtime API calls, we would like to include
the runtime API information in the metadata (for context: #11648).

This feature would help tools like subxt, polkadot-js, and capi provide a uniform API into the runtime.

Furthermore, after merging the RPC Spec V2 API into Substrate, the current RPC methods
calling into the runtime will be replaced by chainHead_unstable_call (the equivalent of the
current state_call method). One example is fetching the metadata via state_getMetadata
that would be replaced by a runtime API call to Metadata_metadata without parameters.

Discovering which runtime function is available, providing the parameters, and interpreting the
result into the proper shape are tedious and prone to error steps. Including the runtime API info
in the metadata would improve the user experience.

The decl_runtime_apis from Substrate will be modified to include the runtime API information in the metadata.

// CC: @jsdw @niklasad1 @bkchr @ascjones @athei @harrysolovay

[xlc]

It will be great if we can avoid breaking changes.
It doesn't have to be breaking changes to expose additional runtime API information. Just add a new API for that.

[lexnv]

Hi,

Thanks @xlc for providing input on this issue.

Regarding exposing multiple flavors of the metadata, we have discussed something similar in issue: #12370.
As Pierre @tomaka mentioned in #12370 (comment):

We don't want to introduce multiple flavors of metadata. This makes everything more complex. Our number one priority in everything we design nowadays should be to reduce the complexity as much as possible.

Although it would be a benefit from the user perspective to not have a breaking change,
adding separate layers/flavors of the metadata increases the complexity and would make
the implementation of paritytech/polkadot-sdk#291 and #10057 more difficult. Considering we had 14 metadata
changes in a relatively short span of time, I would incline towards a V15 release.
Also the next v15 metadata would reflect an ergonomic boost for the end-user.

@bkchr what do you think?

[xlc]

There is no filter required. I am asking to add a new API runtimeApiMetadata in additional to the current metadata
Old SDK simply ignores it. New SDK can use it to ensure the correct type.

I am really against breaking change of metadata because it will break EVERYTHING. Every exchanges, bots, hardware wallet, wallets, dApps, blockexplorer, scripts etc have to upgrade. Many of them are running perfectly otherwise. This is going to cost a lot of dev resources & money.

And what do we gain from this breaking change? Yeah runtime API metadata is super useful. But 99% of the current code doesn't need it and it is just a bad idea to force those code to upgrade for no benefits.

[athei]

I have to agree with @xlc here. Breaking will be too bad for the ecosystem and we should not do it unless absolutely necessary. Additionally, I wouldn't call it metadata "flavors" but versioned metadata. IMHO we should move to version 15 but still provide v14 under its original location.

We should a an additional fallible Metadata_metadata_at_version(Option<u32>) which returns the metadata at the requested version. It can of course fail since we don't support old versions forever. But supporting two versions at the same time is the absolute minimum to allow for a migration.

In this case supporting the old version is not hard as it is just the new version minus the runtime API section.

[bkchr]

We should a an additional fallible Metadata_metadata_at_version(Option<u32>) which returns the metadata at the requested version. It can of course fail since we don't support old versions forever. But supporting two versions at the same time is the absolute minimum to allow for a migration.

Then we can also just introduce metadata v15 format "today" and then wait 3 months until we start returning it from any runtime.

The entire metadata is versioned from almost the beginning and the argument "any change will take quite a lot of dev resources" is a bad argument. This means we could never change anything and this is really bad. We are still soo early and people need to be able to adapt. If we give people 3 months or maybe more to update, it should be possible to adapt all your stuff to the new metadata and have all software updated. I mean we could maybe come up with something like, only one metadata version per year and always collect stuff, but completely stopping is a really bad idea. This would prevent any kind of innovation and I will not support this.

[bkchr]

To what we should add:

1. Tokens: Revamp locking & reserving polkadot-sdk#236

2. Expose types for the overarching Call, Error, Event types frame-metadata#43 V14 metadata doesn't provide enough information to decode extrinsic #12929 Both are going into the same direction. Now with scale-info, we could probably change the format completely. However, while thinking about this, it would maybe prevent any kind of "metadata merkalization".

[athei]

Then we can also just introduce metadata v15 format "today" and then wait 3 months until we start returning it from any runtime.

Why? The old runtime API will still exist. Downstream tools will start using the new runtime API at their own pace. We will remove the old version of the metadata and runtime API once we are convinced that everybody who is willing to keep up has switched.

I am not saying we need to support old stuff forever. I agree that it is too early for that. It is substrate's big advantage that it actually can change. I am merely advocating to allow for a transition period where we have the old and the new one supported at the same time. That will make it much much less of a world breaking event.

[lexnv]

Metadata with Runtime API

Developers add runtime API in the following manner:

1. Define a trait using the decl_runtime_apis macro
2. Implement the API using the impl_runtime_apis macro

The first step contains helpful information as the documentation. At the same time, the second step is where we know which traits (runtime APIs) are implemented for which runtime.

This example declares a Metadata runtime API:

substrate/primitives/api/src/lib.rs

Lines 733 to 737 in ed3f055

	/// The `Metadata` api trait that returns metadata for the runtime.
	pub trait Metadata {
	/// Returns the metadata of a runtime.
	fn metadata() -> OpaqueMetadata;
	}

Users that want to call the metadata function need to make an RPC call (state_call) to the Matadata_metadata function name and provide hex-encoded scale-encoded argument bytes (in this case 0x). The user is expected to interpret the resulting bytes as OpaqueMetadata.

The function name is obtained by concatenating the trait name with the method name (note: there is also an API versioning that we should take into account).

Frame-Metadata

The frame-metadata crate stores the type of metadata and the latest version is v14.

The v15 metadata extends the v14 with the runtime field. This field (similarly to the pallets field) contains all the information needed to generate an API in higher-level projects (subxt, capi etc).

The runtime field is a collection of TraitMetadata objects that contain:

• trait name
• optionally a trait version
• methods available:
  • method name
  • method inputs (name and meta::Type - ie bytes: &[u8])
  • method output (meta::Type)
  • documentation
• documentation (that is only visible at decl_runtime_apis and valuable for UX)

Substrate

The decl_runtime_apis! and impl_runtime_apis! macros are defined in primitives/api/proc-macro.
While the metadata V14 is constructed by frame/support/procedural via the construct_runtime! macro.

All three macros generate crate access: construct_runtime! has access to the frame-support crate (including things like scale-info and frame-metadata) while decl_runtime_apis and impl_runtime_apis have access to the sp-api crate.

The decl_runtime_apis is extended to generate the TraitMetadata for each runtime API.
Although, this could happen in the impl_runtime_apis macro the documentation is no longer available at that point.

For populating the metadata, the decl_runtime_apis needs access to:

• scale_info::TypeInfo: describe runtime types
• frame_metadata::v15::Metadata: to construct the metadata info of the runtime types

Those crates are not available in the sp_api, therefore the crate needs to export scale-info and frame_metadata too.
All the substrate types that are part of a runtime declaration must also implement the scale_info::TypeInfo trait.

The impl_runtime_apis macro implements a method on the runtime to expose the TraitMetadata information collected for each runtime trait implemented. This is similar to the Runtime::metadata() method exposed by the construct_runtime! macro.

To allow a smooth transition for developers and avoid a sudden breaking change in the ecosystem, the Metadata trait is extended with a metadata_v15. The Metadata_metadata runtime function will still return the V14 metadata, while
Metadata_metadata_v15 will return the new enriched metadata.

After a few months, the Metadata_metadata will return the V15 and the Metadata_metadata_v15 is deprecated.

Summary

Frame-metadata:

• Implement Frame-Metadata V15

Substrate:

• Export scale-info and frame-metadata by sp-api
• Derive scale_info::TypeInfo for runtime types
• decl_runtime_apis: generate TraitMetadata for each trait
• impl_runtime_apis: collect TraitMetadata and expose Runtime::runtime_metadata() method
• Extend Metadata trait with Metadata::metadata_v15 method give the community time for transition
• Metadata_metadata method to return: frame-metadata::v15 including runtime API metadata

PoC

• frame-metadata: metadata containing all details about the runtime
• substrate: generate runtime metadata at impl_runtime_apis without documentation
• subxt: use v15 and slim code-gen for the API

// CC: @jsdw @niklasad1 @bkchr @ascjones @athei

[bkchr]

Why? The old runtime API will still exist. Downstream tools will start using the new runtime API at their own pace. We will remove the old version of the metadata and runtime API once we are convinced that everybody who is willing to keep up has switched.

One problem of doing this is that it would prevent of "merklized metadata" where we include the metadata storage root in the signing process to ensure that no one can fake the metadata. If we would have different flavours of the metadata.. While writing this I realized that we could just include both versions of the metadata storage roots in the signing process.

We should a an additional fallible Metadata_metadata_at_version(Option<u32>) which returns the metadata at the requested version. It can of course fail since we don't support old versions forever. But supporting two versions at the same time is the absolute minimum to allow for a migration.

Okay, I'm releasing my blocking on this :P I'm fine with doing this now :P

[athei]

To allow a smooth transition for developers and avoid a sudden breaking change in the ecosystem, the Metadata trait is extended with a metadata_v15. The Metadata_metadata runtime function will still return the V14 metadata, while
Metadata_metadata_v15 will return the new enriched metadata.

Why do not use the more generalized API I proposed? Adding Metadata_metadata_v15 will require new functions every time we update the metadata.

While writing this I realized that we could just include both versions of the metadata storage roots in the signing process.

Not updated clients only know about about the old storage root. So they cannot include both (or all currently valid metadata versions). Instead, a runtime needs to check the signature against all currently supported runtime versions (usually 2).

Okay, I'm releasing my blocking on this :P I'm fine with doing this now :P

Nice

[bkchr]

Not updated clients only know about about the old storage root. So they cannot include both (or all currently valid metadata versions). Instead, a runtime needs to check the signature against all currently supported runtime versions (usually 2).

We could return the metadata storage roots of all supported metadata versions. The client only needs to ensure that one of the storage roots matches the metadata it supports.