Proof generation performance regression since o1js 2.5.0

[youtpout]

Description

Hello,
I noticed a big drop in performance for proof creation on o1js when I upgraded to version 2.5.0 and higher.
With 2.4.0 , I need like 27s to swap from mina and 19s to swap from token
With 2.5.0 , I need like 35s to swap from mina and 25s to swap from token

How to reproduce

https://github.com/Lumina-DEX/lumina-v1-core/tree/fix/improve-multisig

I launch the benchmark on contract folder
Don't forget to pass proofenabled at true in the benchmark.test.ts file
npm run test:benchmark

[Trivo25]

Thanks for bringing this up, we'll look into it!

[querolita]

Update
We have two hypothesis of what could be causing this issue:

• either the lazy mode in o1-labs/proof-systems
• or the updated version of the rayon dependency

We will report back with results of the investigation shortly.

[querolita]

On Ubuntu 24.04 local o1js builds

// ./run ./src/examples/zkprogram/program.ts --bundle

// main (rayon =1.10.0, lazycache yes)
compile (with cache): 12.939s
proving: 12.049s

compile (with cache): 1.746s
proving: 10.700s

compile (with cache): 2.007s
proving: 22.453s

compile (with cache): 3.136s
proving: 26.450s

// main (rayon 1.5.0, lazycache no)
compile (with cache): 2.575s
proving: 22.190s

// main (rayon 1.5.0, lazycache yes)
compile (with cache): 1.916s
proving: 21.444s

// main  (rayon =1.10.0, lazycache no)
compile (with cache): 1.884s
proving: 19.574s

// 2.4.0 (rayon 1.5.0, lazycache no)
compile (with cache): 1.607s
proving: 11.660s

compile (with cache): 1.837s
proving: 8.651s

// 2.5.0 (rayon =1.10.0, lazycache yes)
compile (with cache): 1.791s
proving: 11.115s

// ./run ./src/examples/zkprogram/bench-regression.ts --bundle 

// main (rayon =1.10.0, lazycache yes)
compile (no cache): 37.135s
12289
proving: 19.993s

compile (no cache): 20.033s
12289
proving: 11.575s

compile (no cache): 40.851s
12289
proving: 23.974s

// main (rayon 1.5.0, lazycache no)
compile (no cache): 36.611s
12289
proving: 19.860s

// main (rayon 1.5.0, lazycache yes)
compile (no cache): 37.752s
12289
proving: 20.149s

// main  (rayon =1.10.0, lazycache no)
compile (no cache): 37.443s
12289
proving: 19.966s

// 2.4.0 (rayon 1.5.0, lazycache no)
compile (no cache): 15.139s
12289
proving: 12.949s

compile (no cache): 15.278s
12289
proving: 9.835s

compile (no cache): 17.772s
12289
proving: 12.800s

// 2.5.0 (rayon =1.10.0, lazycache yes)
compile (no cache): 20.022s
12289
proving: 13.670s

// ./run src/examples/zkprogram/bench-regression-long.ts

// v2.4.0
compile (no cache): 46.022s
58509
proving: 22.252s

compile (no cache): 51.011s
58509
proving: 26.815s

[querolita]

Flamegraphs of running medium (12k rows) and long (58k rows) circuits for the following configurations, on Ubuntu 24.04:

Overall, the prover time is observed to have increased by at least 25-30%.

---

v2.4.0: commit 9628358

querolita/bench/v2_4_0

Most of the time is spent while waiting and sleeping on compiled WASM.

Medium circuit: 32 seconds

Long circuit: 82 seconds

------------------------------------------------------------------------------------

v2.5.0: commit 065a15f

querolita/bench/v2_5_0

Most of the time is spent while on caml_pasta_fp/fq_plonk_index_create and caml_pasta_fp/fq_plonk_proof_create

Medium circuit: 39 seconds

Long circuit: 110 seconds

------------------------------------------------------------------------------------

v2.5.0+revert rayon: without commit proof-systems:9cdaa0fa30f6989fb2c1babb9662093d8a2c569f

querolita/bench/v2_5_0_rayon

Medium circuit is shown to spend most of the time sleeping on compiled WASM, whereas long circuit is shown to spend most of the time while on caml_pasta_fp_plonk_proof_index_create and caml_pasta_fp_plonk_proof_create.

Medium circuit: 41 seconds

Long circuit: 107 seconds

---

v2.5.0+revert lazy mode: without commit proof-systems:b2c174582efa40417fa6054b5cc7434d573407b9 and mina:9c42b40d71cf4c58e98b655a44f4d2f70aeadba3 and o1js' b2c1745

querolita/bench/v2_5_0_lazy

Most of the time is spent while on caml_pasta_fp/fq_plonk_index_create and caml_pasta_fp/fq_plonk_proof_create in the medium-sized circuit, whereas in the long circuit it is shown while sleeping on WASM compile.

Medium circuit: 67 seconds

Long circuit: 183 seconds

---

v2.5.0+revert lazy mode+revert rayon: mix of the two above

querolita/bench/v2_5_0_lazy_rayon

Most of the time is spent while on caml_pasta_fp/fq_plonk_index_create and caml_pasta_fp/fq_plonk_proof_create in both circuits.

Medium circuit: 67 seconds

Long circuit: 180 seconds

---

main: 4e78ea0

querolita/bench/main

Most of the time is spent while on caml_pasta_fp/fq_plonk_index_create and caml_pasta_fp/fq_plonk_proof_create in both circuits.

Medium circuit: 71 seconds

Long circuit: 207 seconds

[querolita]

From @Trivo25's macOS M chip, running src/examples/zkprogram/bench-regression-medium.ts

Branch	run #1		run #2
	Compilation in s	Proving in s	Compilation in s	Proving in s
querolita/bench/v2_4_0	11.81s	8.98s	11.84s	8.57s
querolita/bench/v2_5_0	15.45s	10.46s	15.01s	9.96s
querolita/bench/v2_5_0_rayon	14.63s	11.42s	16.72s	10.72s
querolita/bench/v2_5_0_lazy	32.24s	18.91s	33.08s	19.95s
querolita/bench/v2_5_0_lazy_rayon

[youtpout]

From @Trivo25's macOS M chip, running src/examples/zkprogram/bench-regression-medium.ts

The table looks badly formatted

[querolita]

RE: about lazy mode

Some date misalignments made me believe that the lazyMode could be causing this, but actually the lazy mode was not fully added back in 2.5.0 (it was in 2.7.0 indeed). So we must discard this as the cause of the performance drop.

Just for clarity, here's the log of changes of o1js#v2.5.0:

2.5.0 - 2025-05-02

Fixed

• Correct handling of actions and events lists as well as internal
  AccountUpdate consistency tests for the new V2 API.
  Fix V2 AccountUpdate conversion tests #2134

Added

• Improved developer experience and build process.
  replace src/bindings submodule with contents #2076
  • o1js-bindings is no longer a submodule (same directory structure)
  • compiled artifacts are now gitignored
  • npm run build:bindings-download downloads compiled artifacts from github
  • npm run build:bindings-remote triggers a github workflow to build the
    compiled artifacts then downloads them
• Added bitwise operation methods (xor, not, and, or) to UInt8 class.
  Add bitwise methods to UInt8 #2144

Changed

• Bump up Mina to the commit
  5a9145feaba3138cd1a1090d8421a8e67a5485e1
  Bump up Mina to a compatible commit before moving into proof-systems #2126
• Bump up Mina to the commit
  eaca9201,
  to include the latest changes reg. the move of the Rust codebase to the
  repository proof-systems. Cargo: enforce some versions previously used by succesful builds on o1js #2128
• Added verification key validity checks to LocalBlockchain.
  Add Verification Key Validation To Localblockchain #2171

And here's the release notes:

What's Changed

• Release v2.4.0 by @github-actions in Release v2.4.0 #2106
• update changelog job runner by @45930 in update changelog job runner #2116
• Add eddsa to packages by @Trivo25 in Add eddsa to packages #2122
• Bump up bindings by @dannywillems in Bump up bindings #2113
• Nix: update Rust versions by @dannywillems in Nix: update Rust versions #2127
• update node in devshell by @Geometer1729 in update node in devshell #2129
• Bump up Mina to a compatible commit before moving into proof-systems by @dannywillems in Bump up Mina to a compatible commit before moving into proof-systems #2126
• Skip if release exists by @Trivo25 in Skip if release exists #2124
• Cargo: enforce some versions previously used by succesful builds on o1js by @dannywillems in Cargo: enforce some versions previously used by succesful builds on o1js #2128
• Reintro commit 01f043a in proof-systems by @dannywillems in Reintro commit 01f043a in proof-systems #2137
• Reintro commit 76f44f3 in proof-systems by @dannywillems in Reintro commit 76f44f3 in proof-systems #2138
• Reintro commit c103083 in proof-systems by @dannywillems in Reintro commit c103083 in proof-systems #2139
• Reintro commit 8837834 from proof-systems by @dannywillems in Reintro commit 8837834 from proof-systems #2140
• Add bitwise methods to UInt8 by @boray in Add bitwise methods to UInt8 #2144
• MerkleMap: describe what it is. by @dannywillems in MerkleMap: describe what it is. #2153
• Reintro commit af6f5f3 from proof-systems by @dannywillems in Reintro commit af6f5f3 from proof-systems #2141
• Bump up to latest compatible by @dannywillems in Bump up to latest compatible #2158
• Bump bindings to Mina's compatible by @mrmr1993 in Bump bindings to Mina's compatible #2168
• Update doc comments around verify functions by @45930 in Update doc comments around verify functions #2169
• Bump up once_cell to 1.21.3 by @dannywillems in Bump up once_cell to 1.21.3 #2172
• replace src/bindings submodule with contents by @Geometer1729 in replace src/bindings submodule with contents #2076
• Bump up bindings for 0.2.89 wasm-bindgen (copy) by @Trivo25 in Bump up bindings for 0.2.89 wasm-bindgen (copy) #2177
• Mega Typo Fix by @boray in Mega Typo Fix #2178
• Fix V2 AccountUpdate conversion tests by @Trivo25 in Fix V2 AccountUpdate conversion tests #2134
• Reorganize CI + fix ci on main by @Geometer1729 in Reorganize CI + fix ci on main #2180
• Add Verification Key Validation To Localblockchain by @boray in Add Verification Key Validation To Localblockchain #2171
• Add typo check workflow by @boray in Add typo check workflow #2181
• Release v2.5.0 by @45930 in Release v2.5.0 #2191

Full Changelog: 2.4.0...2.5.0

[dfstio]

I also have noticed recently that tiny ZkProgram with 600 constraints takes me 10-11 seconds to prove on Mac M2 Max, while it usually should be less. I have downgraded to o1js 2.4.0 and again see the usual 5-7 sec. I have tested with several o1js versions, and starting from 2.5, I see performance regression.

If I upgrade to 2.7.0, I see one more effect: proving fails with cache. When I run the test first time without cache, everything works, starting from the second time, when cache is available, I see

 Error: Constraint unsatisfied (unreduced):
  File "src/mina/src/lib/pickles/wrap_main.ml", line 514, characters 21-28
  File "src/mina/src/lib/pickles/wrap_main.ml", line 168, characters 17-24
  
  Constraint:
  (Equal(Var 102287)(Constant 0x0000000000000000000000000000000000000000000000000000000000000001))
  Data:
  Equal 0 1
      at failwith (/nix/store/zw9wgfrsagd6sjkw254mvhwhlnd5cj1r-ocaml-base-compiler-4.14.0/lib/ocaml/stdlib.ml:29:14)
      at <anonymous> (/nix/store/596px0jmr4zd511ci22l3l692dhwdjbh-squashed-ocaml-dependencies/lib/ocaml/4.14.0/site-lib/base/printf.ml:6:43)
      at <anonymous> (src/mina/src/lib/snarky/src/base/checked_runner.ml:159:13)
      at run (src/mina/src/lib/snarky/src/base/snark0.ml:737:27)
      at is_true (src/mina/src/lib/snarky/src/base/snark0.ml:870:29)
      at <anonymous> (src/mina/src/lib/pickles/wrap_main.ml:515:15)
      at with_label (src/mina/src/lib/snarky/src/base/snark0.ml:1241:15)
      at <anonymous> (src/mina/src/lib/pickles/wrap_main.ml:514:11)
      at with_label (src/mina/src/lib/snarky/src/base/snark0.ml:1241:15)
      at step_domains (src/mina/src/lib/pickles/wrap_main.ml:167:20)
      at <anonymous> (src/mina/src/lib/concurrency/promise/js/promise.js:27:16)
      at <anonymous> (src/mina/src/lib/pickles/wrap.ml:556:59)
      at handle (src/mina/src/lib/snarky/src/base/snark0.ml:1225:15)
      at _imP_ (src/mina/src/lib/pickles/wrap.ml:556:13)
      at mark_active (src/mina/src/lib/snarky/src/base/snark0.ml:1154:19)
      at _kIH_ (src/mina/src/lib/snarky/src/base/snark0.ml:1338:60)
      at as_stateful (src/mina/src/lib/snarky/src/base/snark0.ml:743:15)
      at _kWd_ (src/mina/src/lib/snarky/src/base/runners.ml:413:17)
      at run_computation (src/mina/src/lib/snarky/src/base/runners.ml:333:34)
      at generate_witness_conv (src/mina/src/lib/snarky/src/base/runners.ml:412:9)
      at <anonymous> (src/mina/src/lib/snarky/src/base/snark0.ml:1338:27)
      at finalize_is_running (src/mina/src/lib/snarky/src/base/snark0.ml:1260:15)
      at generate_witness_conv (src/mina/src/lib/snarky/src/base/snark0.ml:1337:7)
      at <anonymous> (src/mina/src/lib/pickles/wrap.ml:570:27)
      at time (src/mina/src/lib/pickles/common.ml:64:3)
      at max_proofs_verified$0 (src/mina/src/lib/pickles/wrap.ml:520:5)
      at wrap_main (src/mina/src/lib/pickles/wrap.ml:520:5)
      at <anonymous> (src/mina/src/lib/concurrency/promise/js/promise.js:38:29)
      at async withThreadPool (o1js/src/lib/proof-system/workers.ts:60:16)
      at async prettifyStacktracePromise (o1js/src/lib/util/errors.ts:129:12)
      at async prove_ (o1js/src/lib/proof-system/zkprogram.ts:480:18)
      at async TestContext.<anonymous> (/Users/mike/Documents/Silvana/zk-tests/packages/sui/commitment/mina/test/commitment.test.ts:166:28)
      at async Test.run (node:internal/test_runner/test:1069:7)
      at async Suite.processPendingSubtests (node:internal/test_runner/test:752:7)

If I downgrade to any previous version, cache works as usual, I see this problem only on o1js 2.7

It can be tested by running

npm i
npm run test:commitment
npm run test:commitment

in https://github.com/SilvanaOne/zk-tests/tree/main/packages/sui/commitment/mina

[dfstio]

I see that bindings, mina, and wasm-bindgen were changed. Would love to see o1js calling directly rust code without any wasm (i.e., using shared memory, uds, or gRPC over uds) and native rust compiled from proof-systems in a separate process, to forget about 4 GB memory limitations and wasm problems, at least in a server environment.

[querolita]

Deleting querolita/bench/v2_5_0_lazy and querolita/bench/v2_5_0_lazy_rayon to avoid confusion.

Now benchmarking incrementally after each PR merged on top of v2.4.0 (./run src/examples/zkprogram/bench-regression-medium.ts), on Ubuntu 24.04 Intel:

From v2.4.0 up to v2.5.0

Branch	run #1		run #2
	Compilation in s	Proving in s	Compilation in s	Proving in s
9656df2 (v2.4.0)	15.076s	9.378s	15.071s	9.493s
67be3f8 (PR#2116)	15.786s	10.357s	15.371s	10.170s
babd20d (PR#2122)	15.240s	9.651s	14.883s	9.591s
66f698a (PR#2113)	15.182s	9.830s	14.973s	9.467s
705b623 (PR#2127)	15.191s	9.440s	15.055s	9.537s
1678073 (PR#2129)	15.451s	9.780s	15.166s	9.546s
2e365ed (PR#2126)	15.418s	10.133s	15.620s	9.549s
5679460 (PR#2124)	15.583s	9.428s	15.283s	10.406s
b9873e8 (PR#2128) -> dependencies versions update (rayon) ⚠️	20.038s	12.044s	20.308s	12.419s
ace28ce (PR#2137)	20.872s	12.296s	21.044s	12.231s
79f167c (PR#2138)	19.898s	11.838s	20.579s	12.367s
b372e42 (PR#2139)	19.666s	11.495s	19.892s	12.057s
df1ff77 (PR#2140)	19.546s	11.625s	19.723s	11.445s
6afcc06 (PR#2144)	19.901s	11.462s	19.533s	11.449s
7f5057d (PR#2153)	19.410s	11.446s	19.825s	11.459s
b29ad22 (PR#2141)	19.453s	11.479s	19.539s	11.469s
28821f1 (PR#2158)	19.602s	11.723s	19.366s	11.420s
9044f38 (PR#2168)	19.366s	11.420s	19.642s	11.392s
4ac507d (PR#2169)	19.527s	11.396s	19.639s	11.331s
4ca3d17 (PR#2172) -> bump-up once_cell	20.298s	12.414s	19.939s	12.270s
58e6723 (PR#2176) -> replace submodule bindings	19.936s	11.562s	19.624s	11.524s
f274fc0 (PR#2177) -> bump for wasm-bindgen	19.548s	12.481s	19.772s	11.517s
47eff1c (PR#2178)	19.665s	11.913s	19.716s	11.641s
76fd709 (PR#2134)	19.884s	12.087s	19.587s	11.544s
3322c49 (PR#2180)	19.631s	11.652s	19.571s	11.425s
932189d (PR#2171)	19.480s	11.703s	19.538s	11.601s
21d0860 (PR#2181)	19.472s	11.512s	19.707s	11.532s
065a15f (v2.5.0)	19.592s	11.385s	19.535s	11.472s

Then, the same commits but decrementally from v2.5.0 (./run src/examples/zkprogram/bench-regression-medium.ts), on Ubuntu 24.04 Intel, until reaching the bindings replacement:

From v2.5.0 down to v2.4.0

Branch	run #1		run #2
	Compilation in s	Proving in s	Compilation in s	Proving in s
065a15f (v2.5.0)	20.028s	12.075s	21.946s	13.183s
21d0860 (PR#2181)	20.568s	12.306s	20.044s	12.089s
932189d (PR#2171)	20.104s	12.155s	20.190s	12.108s
3322c49 (PR#2180)	20.066s	11.993s	20.127s	12.127s
76fd709 (PR#2134)	20.153s	12.171s	20.517s	11.784s
47eff1c (PR#2178)	20.083s	12.352s	20.373s	12.294s
f274fc0 (PR#2177) -> bump for wasm-bindgen	20.235s	12.416s	20.385s	12.232s
58e6723 (PR#2176) -> replace submodule bindings	20.587s	12.869s	20.272s	12.475s
4ca3d17 (PR#2172) -> bump-up once_cell	20.055s	11.821s	20.334s	11.910s
4ac507d (PR#2169)	20.575s	11.546s	19.991s	11.639s
9044f38 (PR#2168)	19.784s	11.777s	19.977s	11.664s
28821f1 (PR#2158)	19.711s	11.777s	19.809s	11.657s
b29ad22 (PR#2141)	19.810s	11.691s	19.945s	11.659s
7f5057d (PR#2153)	19.866s	12.212s	19.724s	11.624s
6afcc06 (PR#2144)	19.673s	11.497s	19.843s	11.652s
df1ff77 (PR#2140)	19.754s	11.681s	19.874s	11.717s
b372e42 (PR#2139)	20.591s	11.933s	19.917s	11.705s
79f167c (PR#2138)	20.459s	11.497s	21.548s	12.130s
ace28ce (PR#2137)	20.427s	11.633s	20.065s	11.640s
b9873e8 (PR#2128) -> dependencies versions update (rayon) ⚠️	19.718s	11.602s	20.094s	11.639s
5679460 (PR#2124)	15.138s	9.513s	14.966s	9.313s
2e365ed (PR#2126)	14.873s	9.369s	15.008s	9.542s
1678073 (PR#2129)	14.662s	10.865s	14.737s	10.357s
705b623 (PR#2127)	15.591s	10.801s	15.481s	9.976s
66f698a (PR#2113)	15.311s	9.482s	15.321s	9.899s
babd20d (PR#2122)	15.593s	10.093s	15.393s	9.955s
67be3f8 (PR#2116)	15.476s	10.227s	15.541s	10.154s
9656df2 (v2.4.0)	15.703s	10.050s	15.534s	10.634s

These runs confirm consistently that the PR#2128 introduced a performance drop of about 25% in both compilation and proving times.

[querolita]

Now inspecting the changes introduced in #2128.

• The branch querolita/bench/main_revert_rayon aims at obtaining a version of o1js but reverting the rayon version update introduced in that PR. Meaning, it just goes back from =1.10.0 to 1.5.0, and revendors dependencies.
• The branch querolita/bench/main_revert_kimchi_3144 aims at obtaining a version of o1njs but reverting the packages versioning introduced in the proof-systems counterpart of that PR (meaning Cargo: enforce some versions previously used by succesful builds on o1js proof-systems#3144).

In both cases, the performance obtained after running the medium-sized circuit (12k rows) is even worse in my Ubuntu 24.04 (but last week when running it, it seemed faster i.e. 36s vs 40s, so take it with a grain of salt)

Branch	run #1		run #2
	Compilation in s	Proving in s	Compilation in s	Proving in s
9656df2 (v2.4.0)	15.703s	10.050s	15.534s	10.634s
b9873e8 (PR#2128) -> dependencies versions update ⚠️	19.718s	11.602s	20.094s	11.639s
065a15f (v2.5.0)	20.028s	12.075s	21.946s	13.183s
2e5e9c7 (main) ⚠️	37.987s	37.987s	37.326s	20.801s
querolita/bench/main_revert_rayon	40.432s	24.575s	40.712s	24.000s
querolita/bench/main_revert_kimchi_3144	40.655s	25.974s	41.694s	23.893s
querolita/bench/v250_revert_kimchi_3144	36.721s	20.294s	37.377s	20.358

This suggests that the performance drop could be derived from other changes introduced in the mina bump-up of #2128 (meaning something that happened during MinaProtocol/mina#16856).