Adopt the E2EE cryptography module from matrix-rust-sdk: "Element R"

[novocaine]

Context

Element Web currently uses a transpiled version of https://gitlab.matrix.org/matrix-org/olm originally written in C++

Using the rust-sdk's WASM bindings, we replace our entire crypto implementation for performance, stability, security, and maintenance reasons. This project is already approved for work to begin.

The intent is to support both old & new crypto implementations side-by-side through a labs flag up until we're ready to ship.

Update March 2024

Element Web and Element Desktop have been using the Rust crypto stack for new logins for several weeks now. All users on the develop.element.io instance of Element Web have also been migrated to the Rust stack.

Next step, hopefully starting this week (w/c 25 March 2024) is to roll out that migration to app.element.io users.

Update January 2024

We've been continuing to work on defects and missing features in Element Web R.

We've also developed a "migration" process, which will take a session using legacy crypto, and convert it to rust crypto. Once #26772 lands, you will be able to test it out by enabling the "Rust Cryptography" switch in the "labs" settings.

The next stage is to enable it for new logins on "stable" deployments (such as app.element.io, and Element Desktop). The things remaining to be sorted before that can happen are tracked at https://github.com/element-hq/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker.

Update October 2023

https://develop.element.io has been updated so that new logins will use the rust cryptography implementation.

Update June 2023

We are now using a project board to track this work: https://github.com/orgs/vector-im/projects/76/views/15. "Prioritized Backlog" represents the work we consider to be required for the "MVP" of enabling on https://develop.element.io.

To try out the work-in-progress, see matrix-org/matrix-react-sdk#10080.

Plan from December 2022

High level overview of approach

(as of December 2022)

• 1. Expose a config.json setting which will make the js-sdk use matrix-sdk-crypto-js instead of libolm. At this stage there will be no support for migrating existing sessions between libolm and rust.
• 2. Build up functionality based on the rust sdk:
  • basic encryption and decryption support
  • attachment/media support
  • Key backup
  • Cross signing
  • Key sharing
    ... plus other things TBD.
• 3. Implement a one-way migration from libolm to rust, and allow people to opt into it via the labs flag.
• 4. Further work to achieve parity with the libolm implementation. (Matrix Content Scanner integration? Extensions etc for Element Call and widgets?)
• 5. Finish migration:
  • Enable the labs flag by default for new sessions
  • Forcibly migrate existing sessions to rust sdk
  • Get rid of the config flag and legacy code

After step 2, Element R web should be good enough to replace libolm-based Element Web as a daily driver. We'll begin to encourage users to try out a hosted instance of it and give feedback, while continuing with step 3 and beyond.

Detailed plans from April 2022

Phases

Phasing is approximate and used as an estimation tool. This list will be updated often - check back frequently.

Time estimates are not guarantees or even accurate - they are relative. They will also be updated as needed.

Phase 1: Prove it works (1-2 months)

• [--] Bindings exist on NPM (in progress)
• [ 5] Introduce labs flag for js-sdk
• [13] Hook up rust stores to storage. Rudimentary one-way migration.
• [13] Hook up to sync loop, basic encryption and decryption support.
  • Including attachment/media support
• [ 8] Key backup (protocol level, not UI)
• [ 8] Key sharing (protocol level, not UI)
• [13] Cross-signing (protocol level, not UI)
• [21] Two-way migration for labs flag, allowing users to "switch" between implementations
  • Potential to implement this as an implementation which writes to both stores saving us from migration

Phase 2: Establish trust in the plan (1-2 months)

• [ 8] Benchmark old crypto & new crypto to establish performance targets
• [??] Tests which prove old crypto was working
  • "Working" is defined as behavioural traits, not necessarily bug-free.
• [??] Tests which prove the new crypto isn't any more broken than old

Phase 3: Build a plausible client (2-4 months with questionable accuracy)

• [??] Self verification
• [??] Other user verification
• [??] Visual indicators for user trust (cross-signing, 4S)
• [??] UI bits for key backup, key sharing, and cross-signing
• [??] Manual key export
• [??] Historical key sharing (for room history)
• [??] Device management

Phase 4: Polish and remaining bits (1-2 months)

• [13] Support for customisation endpoints/modules as needed
• [13] Matrix Content Scanner integration (if needed)
• [??] Device dehydration
• [??] Functions required by Element Call and widgets (custom to-device messages?)
• [??] Posthog metrics
• [??] TBD stuff from https://github.com/matrix-org/matrix-rust-sdk/milestone/1
• [??] Get design involved for migration experience

Phase 5: Stability (1-2 months)

• [??] Enable by default on Nightly and Develop (not production, EMS, app, or staging)
• [??] Fix bugs & build comfort
• [??] Enable by default in production (EMS, app, and staging alongside existing develop channels)
• [??] Fix bugs & build comfort

Phase 6: Release (1 month)

• [??] Convert migration to a one-way migration
• [??] Remove old crypto code (keep migrator)
• [??] Remove labs flag
• [??] Eternal maintenance

Internal references

myhours: https://app.myhours.com/#/projects/1943744/overview

[turt2live]

waiting on poc to prove this is feasible, then will formulate and enact a plan

[novocaine]

@Hywan are there some sub issues or PRs that we can link to this epic?

[turt2live]

matrix-org/matrix-rust-sdk#675 covers the WASM-specific bindings. Crypto team is currently working on a simple encrypt/decrypt demo that will need adapting into production-safe code (which is my problem).

[Hywan]

@novocaine Nothing I'm aware of except the one mentionned by @turt2live hereinabove.

[turt2live]

Rough tasks that will need breaking out to useful issues (once further narrowed down):

• Look at POC
• Add labs flag for dual stack crypto
• Consider what migration UX might look like: eg Database migration UX element-meta#263
• Ongoing: perform benchmarks to see if crypto speed is improving (secondary objective)
• Get basic encryption working
• Get verification working
• Get cross-signing/SSSS working
• Get backup working (might be related to SSSS these days?)
• Figure out what was missed and get it working
• Become comfortable with Rust as a crypto engine through unit tests, integration tests, end-to-end tests, and manual/community testing
• Remove labs flag, switch to only Rust
• Remove legacy code (once sufficient adoption for migration)