bug(`forge test`): fuzzy parameters not generated deterministically from seed

[guidanoli]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

nightly-33cb8d2faf6787ddecde2af7343ee472bdf829ee

What version of Foundryup are you on?

0.3.3

What command(s) is the bug in?

forge test

Operating System

Linux

Describe the bug

Fuzzy parameters are not generated deterministically from the seed.

Steps to reproduce

1. Create a new blank Forge project

pushd "$(mktemp -d)"
forge init .

2. Create a new test file with the following contents. It contains a single test case which writes all the fuzzy parameters to the same unique file.

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import {Test} from "forge-std/Test.sol";

contract FuzzyParamsTest is Test {
    string dumpFilePath;

    function setUp() external {
        for (uint256 i; ; ++i) {
            dumpFilePath = string.concat("./fuzzy-", vm.toString(i), ".txt");
            if (!vm.exists(dumpFilePath)) {
                break;
            }
        }
    }

    function testFuzzyParams(uint256[] calldata x) external {
        for (uint256 i; i < x.length; ++i) {
            vm.writeLine(dumpFilePath, vm.toString(x[i]));
        }
    }
}

3. Add the following line to the foundry.toml config file:

fs_permissions = [{ path = ".", access = "read-write" }]

4. Run the test multiple times with the same fuzzy seed.

for n in $(seq 10); do forge test --fuzz-seed 0; done

5. Compare the results by hashing the generated files.

sha256sum fuzzy-*.txt

6. You should observe different hashes. For example:

24f5641b121c32c7dceb41336f7316c17108156aea6229b3bd2efc3505b49d26  fuzzy-0.txt
1527b01b01228e9440374d6db77af00be5acb34419c9ff593990feb6db576ce4  fuzzy-1.txt
3fe220fe544c6b37d13377ad4cdd23ff9521a471ff80855051d131f073ad96e3  fuzzy-2.txt
eff2fb96c10ce4b5bfa55f374ea87d437030174b4efafffeb29004018405661e  fuzzy-3.txt
6e60ac5afee8d3670342f19788fc67ccf92fc8c5ec416ac05bb246eb929dac8a  fuzzy-4.txt
e2ed16785ea787c1b7803fa5e589a3f110e09cc0b7295ff637816c337c73608c  fuzzy-5.txt
ec31b183018d262b35e516e479f4ca8ff6c32c37d8732e0d97eec2fa46e618b0  fuzzy-6.txt
a34f7ce152d7714326cbc616a529fc9090814392beaf760cc10f80f741343b04  fuzzy-7.txt
d220c06b2852ec80a2422b135afe571feee4771ab51dcd82d9d803265dbd4f8b  fuzzy-8.txt
a502f0ae7201f60112ca40782094b4ed29754ddcfa35a7cf8cd6b3595f7d8b65  fuzzy-9.txt

7. Taking a peek, you can see they differ in a quite predictable way.

alias difftool='vimdiff' # Vim
alias difftool='nvim -d' # NeoVim
alias difftool='code -d' # VS Code
difftool fuzzy-0.txt fuzzy-1.txt

8. You should see something like the following.

--- fuzzy-0.txt	2025-05-05 17:36:54.243366075 -0300
+++ fuzzy-1.txt	2025-05-05 17:36:55.219303271 -0300
@@ -215,7 +215,7 @@
 6616
 4318
 7895
-20890139992214452818511113740937997599324735510181082302925590787528146812953
+20890139992214452818535633669591851821058469062615487249863490613483084447769
 5754
 6011
 388
@@ -678,7 +678,7 @@
 5824
 4217
 6969
-20890139992214452818511113740937997599324735510181082302925590787528146812954
+20890139992214452818535633669591851821058469062615487249863490613483084447770
 6589
 3335
 4386
@@ -2521,7 +2521,7 @@
 1830
 46308022326495007027972728677917914892729792999299745830475596687180801507327
 7525
-20890139992214452818511113740937997599324735510181082302925590787528146812954
+20890139992214452818535633669591851821058469062615487249863490613483084447770
 26959946667150639794667015087019630673637144422540572481103610249215
 4188
 177362149

[guidanoli]

This issue is important because it hinders deterministic gas reports, which are important to assess improvements in gas optimizations, etc.

[guidanoli]

Note: this only happens for functions with fuzzy array parameters (T[] and bytes).

[guidanoli]

Also, when I slightly change the source code of the test script, the fuzzy parameters also change.

[grandizzy]

@guidanoli thank you, I am unable to reproduce the issue but with a slightly change to make sure same test / bytecode is used, that is test below

    function testFuzzyParams(uint256[] calldata x) external {
        for (uint256 i; i < x.length; ++i) {
            vm.writeLine("fuzzy.txt", vm.toString(x[i]));
        }
    }

and run forge test --mt testFuzzyParams --fuzz-seed 0. Then mv fuzzy.txt fuzzy1.txt and rerun test. If then you do diff fuzzy.txt fuzzy1.txt they show no difference, and so on. Mind that the contract bytecodes are also used to fuzz so any change will result in different inputs. You could look into using turning that off by https://book.getfoundry.sh/reference/config/testing#include_push_bytes.

[guidanoli]

I didn't know the fuzzer obtained entropy from push bytes and storage values.
These are quite useful sources of entropy.
I reckon I'm misusing fuzzy parameters for gas reports.
To measure gas cost improvements, I should explicitly set out test cases.

[grandizzy]

@guidanoli yeah, we just improved that to avoid metadata (that is non relevant changes like comments to not affect gas results #10402) and we're looking to improve fuzz dict with #10233

Do you think there's anything else to be addresses in this issue or can be closed for now? Thank you