ink! 2.0 Roadmap

[Robbepop]

ink! 2.0 Roadmap

This is a post describing the current state of ink! and its problems and how we are planning to deal with them in the near future.

---

During development of ink! we found several issues in its design and implementation.
These issues, some of them more critical than other, hinder us in delivering the best developer experience possible for smart contract development.

Problem #1: Big-Fat-Macro

In its current form the entire code of an ink! smart contract lives within a macro body call to the contract! macro. On the one hand side this beautifully encapsulates and separates ink! code from Rust code and provides the ink! "compiler" with the ability to reason about everything within. On the other hand side the approach to deal with all smart contract code through a single macro invocation has several downsides:

• RLS and rust-analyzer are having a very hard time reasoning about the internals of macro instantiations and thus we cannot deliver upon our principal of IDE friendliness for ink! at the moment.
• Having all code specified within a single macro body also means that the entire code of a smart contract needs to live within these boundaries which excludes multi-file projects and general coding structure. This is critical to our principle of having a well scaling developer experience for writing smart contracts.

Solution

We are going to solve this by re-implementing ink! 2.0 as a rustc plugin instead of a procedural macro. The main benefits compared to a procedural macro are that a rustc plugin is able to introspect the entirety of the macro expansion invocations and thus can naturally reason about all of the expanded code which is exactly why we introduced the Big-Fat-Macro in the first place. The downside of this solution is that we cannot expect to work on stable Rust anytime soon but we do not consider this a show stopper since nightly Rust has shown to be stable and robust enough for our purposes.

Note: This change will break compatibility with existing smart contracts. Adjustments are going to be necessary.

Problem #2: Types

Accessing types in procedural macros and also in rustc plugins is simply put not possible. This is for a reason and probably won't change. We normally don't need type information when it comes to simple macro expansion rules, however, for ink! we do more than normal macros and for some tasks, such as the metadata (ABI) file generation, we need access to the underlying types specified within the smart contract code.

For this purpose we already put some work into creating a so-called two-phase compilation approach. This will change the project structure of current ink! smart contracts a bit in order to work. Don't worry, out cargo-contract plugin will deal with this upon creation of a new smart contract. Also we might think about a new routine in order to keep project structures up-to-date through a simple call to the cargo-contract plugin.

On the surface for users generating ABI files will no longer be a separate result of a simple compilation of a smart contract but will be its own phase. In order to create ABI files user will have to invoke the cargo-contract plugin with something in the lines of cargo contract generate-abi and ABI files for the smart contract will be created in the target folder as is done today.

Note: This change will break compatibility with existing smart contracts. Adjustments are going to be necessary.

Problem #3: Calling other Smart Contracts

The fundamental feature to call other ink! smart contracts has not yet been realized due to the current problems stated above. Smart contracts need access to the meta information of the smart contracts they depend on in order to guide instantiations and calls of remote smart contracts.
Also having to specify links to other smart contracts in the Big-Fat-Macro world could potentially lead to several other design complications.

So this builds on the availability on the solutions for problems #1 and #2.

What we do not want is low-level calls to primitives such as ext_call that are available today.* (* After merging #133)
What we do want instead is having our rustc plugin create code on the fly for simple instantiation and calling of other smart contracts that are types safe and eventually just feel like calling yet another Rust function. However, due to complications with the fundamental approach and gas costs a proper UX design has yet to be found.

Problem #4: A world full of fees

When ink! was originally developed it was not clear (at least not to me) that state rent will be a thing in the future. As of the current Substrate implementation it now is. This means that smart contract developers need to have this additional piece of complexity in their heads while developing smart contracts.

In essence rental fees mean that a smart contract now has to pay fees in respect to its used storage size. Also there are going to be techniques to distribute storage fees, such as systems similar to cookie-contracts. In theory we could declare ink! to be independent from all this and won't help users design their smart contract economy friendly with regards to state rent. On the other side we think that if we want to make state rent a success, we need to guide users in these new lands of smart contract development. The funny side about this story is that we need to collect experience with these techniques ourselves before we can actually give great advise.

The hopes are the ink! 2.0, once done, won't have to adjust a lot in order to provide guidelines and help with respect to support smart contract developers in developing rental state friendly smart contracts.

Enhancement: Standard Lib

Currently the ink! standard lib (namely ink_core) is pretty small and doesn't feature a whole lot of collections and utilities. This is for a reason since we currently have no option to dynamically link compiled smart contracts and thus have to rely on static linking which blows our Wasm blobs into gigantic bubbles of ones and zeros.

However, the future looks dynamic, and for the sake and purpose of productivity we want to do two things:

• Provide many utilities and collections out of the box for developing ink! smart contracts through a standardized set of libraries. Right now we are missing fundamental collections such as storage::BTreeMap/Set and storage::BinaryHeap to name a few. Especially in very performance constrained smart contract ecosystems it is very important to provide many specialized data structures that are heavily optimized for their use cases in order to save up gas and relieve the chain under pressure.
• Encourage users to provide libraries on their behalf. With the current ink! and its very constrained model it is not very encouraging to develop smart contract libraries. While it is possible, nobody does it and most of the developers develop in isolation. Especially with the solutions of the first two mentioned problems we want to build up ink! 2.0 in a way that encourages for library and interface development in a similar vein as Rust already successfully does.

[Demi-Marie]

We are going to solve this by re-implementing ink! 2.0 as a rustc plugin instead of a procedural macro. The main benefits compared to a procedural macro are that a rustc plugin is able to introspect the entirety of the macro expansion invocations and thus can naturally reason about all of the expanded code which is exactly why we introduced the Big-Fat-Macro in the first place. The downside of this solution is that we cannot expect to work on stable Rust anytime soon but we do not consider this a show stopper since nightly Rust has shown to be stable and robust enough for our purposes.

I strongly advise against this. Rust issues CVEs for bugs in stable Rust that could lead to security vulnerabilities in user code, but does not do so for nightly. Pinging @kirushik.

[kirushik]

@demimarie-parity AFAIU, we're only targeting nightly Rust in Ink anyway.
Do you know of any instances of Rust having a CVE-related fix in nightly, while having it in stable? (Short of de-stabilizing some features which turned out to not be safe enough, like the recent Error::type_idone; I don't see it as an issue — you need to opt in into using every noghtly-only feature anyway, and should be aware of the security implications.)

Otherwise "always try to build with the latest nightly, and explicitly check for updates for the toolchain after there's been a CVE announced for stable" seems to be a reasonable enough policy.

[dvc94ch]

I'm also strongly against this. Rust plugin support may be dropped in the future. rust-lang/rust#29597

You are specifically looking for procedural attribute macros. Though you'd use #![profile_every_procedure] (note the !) to make sure it applies to the entire module and not just the next function.

Though I think the way you are describing it, you'd want this in the crate root and process the entire crate at once. This way you have a global view on the entire crate and can do a global analysis (wrt generating structs only once and such).

[Robbepop]

Thank you for the idea about profile_every_procedure.
It is a neat idea but will only provide us with the vision about a single module.
I had this idea before we concluded the current ink! syntax and back at that time the feature custom_attributes and custom_inner_attributes where too unstable for usage - maybe worth a try now though ...

Another problem back then was that this #![profile_every_procedure] was not working in the crate root lib.rs and caused arcane problems there so you still have to do another encapsulating module for that like so:

File: lib.rs

// some crate imports

#[profile_every_procedure]
mod my_smart_contract {
    // smart contract contents
}

When I tried out #![profile_every_procedure] on the lib.rs level it gave me some arcane magically expanded hidden Rust code to work with and expected some weird expanded code from the macro expansion that was not going to work. The experiement was at around Dec 2018. So maybe it is worth a try again, especially if people like oli_obk mention them.

Also: Maybe this is no longer that broken as it was back then.

Some compile errors I get when enabling custom_attribute and custom_inner_attribute in ink_lang:

error: inconsistent resolution for a macro: first custom attribute, then derive helper attribute
   --> lang/src/api.rs:313:3
    |
313 | #[serde(transparent)]
    |   ^^^^^

error: cannot determine resolution for the derive macro `Serialize`
   --> lang/src/api.rs:332:45
    |
332 | #[derive(Debug, PartialEq, Eq, Deserialize, Serialize)]
    |                                             ^^^^^^^^^
    |

[Demi-Marie]

@Robbepop I would consider reporting the problems as rustc bugs.

[Robbepop]

@Robbepop I would consider reporting the problems as rustc bugs.

In this case there are no bugs but simply open questions since nobody was yet able to find a proper answer towards inner-custom attribute macros and their scope. It is a non-trivial question tbh and I do not know a best solution either.

[flyq]

how about ink! 2.0's schedule now?

[Robbepop]

how about ink! 2.0's schedule now?

The ABI work item is nearly done: #104
The next two most important work items are ink! plugin and calling of remote contracts.
The are going to implement ink! plugin in a two-way fashion alongside the current ink! language (ink_lang) so people are going to be able to use both of them or transit from one to the other with ease.
Calling of remote contracts is a pretty significant feature and I think we should put energy and time into crafting it. Andrew and me are discussing several ideas and will collect them in the associated tracking issue.
It has yet to be decided whether the next goal is calling remote contracts or ink! plugin - the best would be if we could develop both features in parallel.

A very rough estimate is about 2-3 months "man-power" for each.

[flyq]

Awesome!!

[jadbox]

2020 update?
Also does master branch currently require nightly Rust?

[Robbepop]

This roadmap is heavily outdated. Most of its items have already been implemented.
We are going to create a new 2020 ink! roadmap.

Unfortunately ink! does not yet work with stable Rust, so nightly is still required. However, we hope that we can switch to stable Rust soon.

[jadbox]

@Robbepop thanks for the update- do you have a sense on what's required for it to switch to Stable and what features currently require Nightly? It's a little terrifying to speculate using something for contract building that is both experimental and relies on Rust Nightly. heh

[Robbepop]

I just wrote this into our Riot ink! channel:

... We are currently depending on const_fn nightly feature and the macro hygiene nightly feature. However, the macro hygiene nightly feature dependency is going to be dropped in the next (or next-next) stable Rust version since the subset of the feature that ink! is depending on is going to be stabilized.

[In addition] We might be able to get rid of our const_fn feature dependency as well with some additional work.

As soon as Substrate 2.0 is released (which is imminent) we are also planning to go to a proper release cycle for ink!.

[Robbepop]

The ink! 2.0 roadmap lies behind us and we achieved most of the goals stated here in some way or another. Our new focus should be at ink! 2.1 and a roadmap for that shall be created soon. Closed.