The versions of the primitive packages are not synced with crates.io

[clearloop]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

sp-api has version 4.0.0-dev in the git repo https://github.com/paritytech/polkadot-sdk/blob/master/substrate/primitives/api/Cargo.toml#L3

however, it has already been bumped to 25.0.0 in crates.io ?

ref #2808

Steps to reproduce

No response

[DragonDev1906]

I've just come across this, too:
sp_core_hashing has version 9.0.0 in the git repo: but crates.io has been bumped to 14.0.0 already.

[ggwpez]

Yea we currently dont sync back the versions. I have asked about this a few times as well, since it makes things easier. Now with externals asking maybe there is a point cc @Morganamilo

[DragonDev1906]

It is mostly confusing, especially since there is no overview of what has changed (mainly speaking about breaking changes) and (almost) every update seems to be a major version update. Even though I don't think much has changed in basic crates like sp_core_hashing.

Personally I don't think doing a major version bump on a primtive dependency just for having the same starting number is worth making semantic versioning basically useless, but that's a different topic.

[ggwpez]

Personally I don't think doing a major version bump on a primtive dependency just for having the same starting number is worth making semantic versioning basically useless, but that's a different topic.

Yes absolutely. We are working on it here #2490. It is just our "best effort" until we have that.

[Morganamilo]

Yeah the current situation is not exactly ideal. The original version split stems from Polkadot v1.0.0 releasing and us wanting v1.0.0 as the version in the binary releases. Then when v1.1.0 released, wanting v1.1.0 as the version for that binary release and so on.

This is obviously fundamentally incompatible with crates.io's semver policy as these release do have breaking changes on the library side. Also there were existing crate version published with versions much higher than v1.x.

I'll check with the devs and see If there's support for using the crates.io versions also in the git repo. This may be better for consistency but some people may not like the versions no longer following the release versions.

Personally I don't think doing a major version bump on a primtive dependency just for having the same starting number is worth making semantic versioning basically useless, but that's a different topic.

We're hoping to fix this within the next couple of releases. Bumping the major ver every time is done purely because it is guaranteed to be semver safe and there's currently no procedure+tooling to handle semver across the 100s of crates.

My focus before this has been diffing and not publishing crates that have not changed (and more smartly, have not changed semantically). This doesn't solve the major bump issue but should help as if a crate doesn't change we get no bump at all.

I'm also working on tooling to manage figuring out what version to add as a user so the vast array of versions don't confuse.

Can I ask if this issue is purely an observation or has caused issues on your end? As currently I see this disparity as just a bit ugly and perhaps a cause for confusion. I'd be very interested to know if this has lead to actual Issues for people.

[DragonDev1906]

For me only confusion (because I thought the upload to crates.io was not from this repo).

The biggest issues I can see, are:

• Projects depending on these crates not updating to the newest version early because there are so many breaking changes and it doesn't happen automatically as it does with minor semver changes. For example, if you look at https://crates.io/crates/sp-core-hashing more than half of the projects use an old version that is only shown as "others" in the graph (though that's about the fast breaking changes and not about the lack of sync with the version in this repo.
• having to look into the git history (or trying for good luck if it compiles and runs) to see if there is a breaking change that could affect us. Which you have to do for other dependencies, too, but those have far fewer breaking changes and those are usually described somewhere.

[xlc]

polkadot-sdk releases should be decoupled with polkadot releases.

Current approach is not "just a bit ugly". It is completely unbearable. Try to submit a PR to upgrade ORML to use 1.4.0 and you will understand.

I will say just forgot about semvar until polkadot-sdk is refactored in such way that it is actually feasible to have semvar, which I don't think it is ever going to happen.

[ggwpez]

I will say just forgot about semvar until polkadot-sdk is refactored in such way that it is actually feasible to have semvar, which I don't think it is ever going to happen.

Why do you think that per-crate SemVer would not work @xlc ?
Currently the plan is to version each crate according to SemVer and bump the ones that re-export stuff as well. This should allow proper usage of cargo update. Its not enough you think?

PS: we will encounter the same issues in the runtimes repo polkadot-fellows/runtimes#137, so it should help for finding a good solution.

[xlc]

1. There is a risk of using pallets from different releases and there is basically no upside of doing that.
2. It is too easy to introduce a breaking change. Any modification to storages and trait config will be a breaking change. It is rarely the case that we can introduce a new feature without breaking change.
3. There are some work was done to improve it but all the work are disorganized and takes forever. This is a super complicated task and requires many people to work together for many months. It is simply not ever going to be completed without some proper management.

Are there any big Rust projects that are also mono repo? I did a quick search but didn't find anything similar. Maybe this is a solved problem and we should try to take similar approach as other existing projects?

[ggwpez]

1. There is a risk of using pallets from different releases and there is basically no upside of doing that.

Yea, that is an issue. I will try to do some research if this is a solved problem. Otherwise we could publish a "bundle" polkadot-sdk crate that locks in dependency exact and the set all other versions to version = "*".
Its quite unusual to do that i think, but then we would not build stuff twice and can update all at once with cargo upgrade -p polkadot-sdk.
Or we make a tool that only pulls the versions from the bundle crate and scans the Cargo.lock to deny duplicates.

2. It is too easy to introduce a breaking change. Any modification to storages and trait config will be a breaking change. It is rarely the case that we can introduce a new feature without breaking change.

Yes, but we want to have an LTS duration of 3 months for now and all breaking changes in the meantime will not hit that LTS branch. They only go on master. I think we have to try at least. There is some tooling to help with spotting breaking changes.

[wischli]

As a parachain developer, I would be interested in how to map a pallet's semver version from crates.io to the corresponding polkadot-sdk release. I don't hope the best approach is to evaluate this by publishing dates.

[acatangiu]

"Easiest" way I know of doing it is to look at the crate's Cargo.toml on the polkadot-sdk release branches to match versions - but I can first-hand confirm it's a bad UX.

In the future, we could automate adding a description like "released as part of https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-vX.Y.Z" or similar during crate publishing - that should be fairly easy to do while providing considerable pain relief.