Skip to content

test: testify/mock race-detector overhead causes ~13min CPU time in controller/artifact #100

@Vad1mo

Description

@Vad1mo

Problem

The controller/artifact test package consumes ~13 minutes of CPU time when running with -race, despite containing only 36 test functions with no sleeps, no network calls, and no goroutines. The entire overhead comes from testify's mock.Mock synchronization amplified by Go's race detector.

Root cause: mock.Mock (testify v1.11.1) uses a shared sync.Mutex. The MethodCalled() method — invoked on every mock call — performs 4 separate lock/unlock cycles per invocation:

  1. Match expectations + append to Calls slice
  2. Read PanicMsg
  3. Read RunFn
  4. Read ReturnArguments

Go's race detector instruments every mutex operation, turning microsecond locks into millisecond-scale operations.

In controller/artifact: SetupTest() creates 11 mock objects for every one of the 19 suite test methods (209 mock instances per suite run). With 134+ .On() registrations and hundreds of mock calls, the race-detector overhead compounds to ~13 minutes.

This is a known upstream issue — testify #1597 documents data races in Arguments.Diff. Multiple fix PRs (#1598, #1693) remain open. testify has no configuration to disable synchronization or call tracking.

Note: Only testify/mock has this problem. testify/suite and testify/assert have no synchronization overhead and can remain regardless of mock framework choice.

Scope

Not limited to controller/artifact. 33 test files use both testify/suite and testify/mock:

Rank Package Mock count
1 controller/scan 12
2 controller/artifact 11
3 pkg/scan/vulnerability 9
4 pkg/scan/sbom 8
5 server/v2.0/handler/scan_all 7

The project has 93 interfaces configured in .mockery.yaml, generating 105 mock files (21,508 lines) in testing/.

Current testify/mock feature usage

Feature Occurrences Notes
.On().Return() 1,301 Core pattern
mock.Anything 1,081 Wildcard matcher
.Once() 573 Call count = 1
.AssertExpectations() 268 Verify all expected calls
.Times(n) 83 Call count = n
.AssertCalled() 80 Verify specific call
.AssertNotCalled() 40 Verify no call
mock.AnythingOfType() 34 Type matcher
mock.MatchedBy(fn) 32 Custom matcher
.Run(func) 20 Callback on call
.After(duration) 7 Delayed return
.AssertNumberOfCalls() 5 Verify exact count
.Panic() 1 Panic simulation
.Maybe() 0 Not used
.Unset() 0 Not used

Any replacement must cover at minimum: .On().Return(), .Once()/.Times(), mock.Anything, .AssertExpectations(), .AssertCalled()/.AssertNotCalled(), and mock.MatchedBy().

Options

Option 1: Lazy mock initialization in SetupTest

Only create the mocks each test actually uses instead of all 11 in every SetupTest().

Pros Cons
No new dependencies or tooling changes Still uses testify/mock — per-mock overhead unchanged
Minimal code change Must manually track which tests need which mocks
Keeps existing testify/mockery workflow Diminishing returns as test complexity grows

Option 2: Switch to mockery matryer template

mockery (the project's existing mock generator) supports generating moq-style mocks via template: matryer. These generate structs with function fields instead of synchronized maps — zero mutexes, zero race-detector overhead.

The project's .mockery.yaml already uses the packages config format. mockery provides mockery migrate for automated config conversion to v3. v2 is supported through December 2029.

Feature mapping from testify/mock to matryer-style:

testify/mock matryer equivalent
.On("Get", args).Return(val) mock.GetFunc = func(...) { return val }
mock.Anything Not needed — function fields are type-safe
.Once() / .Times(n) len(mock.GetCalls()) == 1 / == n
.AssertExpectations() No direct equivalent — must assert each call
.AssertCalled() len(mock.GetCalls()) > 0
.AssertNotCalled() len(mock.GetCalls()) == 0
mock.MatchedBy(fn) Logic moves into function body
.Run(func(args)) Built-in — function field IS the callback

template-data options: stub-impl (return zero values for unset methods), with-resets (reset call tracking), skip-ensure (avoid import cycles).

Pros Cons
Zero synchronization overhead Requires mockery v2 → v3 upgrade
Same mockery tooling, same go generate Different test assertion style (closures vs .On().Return())
Compile-time type safety No .AssertExpectations() — must assert each call manually (268 uses)
Incremental migration (per-package template: override) .Once() / .Times() replaced by manual call count checks (656 uses)
Automated config migration (mockery migrate)
testify/suite and testify/assert remain unchanged

mockery: 7.1k stars, v3.7.0 (March 2026), actively maintained.

Option 3: moq directly

moq generates function-field mocks. Same concept as Option 2's matryer template, but as a standalone tool.

Pros Cons
Zero synchronization overhead Introduces a second code generator alongside mockery
Simple, focused tool Different CLI and config from mockery
Compile-time type safety No automated migration from testify

moq: 2.2k stars, v0.7.1 (March 2025), last commit February 2025.

Option 4: uber-go/mock (gomock)

uber-go/mock uses controller-based expectations with type-safe generated recorders. See comparison for full feature matrix.

Pros Cons
Type-safe expectations (no string method names) Still has synchronization overhead (less than testify)
Built-in call ordering (InOrder(), After()) Significant API differences — high migration effort
Strict/flexible call verification modes Different generator (mockgen), different config
Actively maintained by Uber Requires controller lifecycle management

uber-go/mock: 3.3k stars, v0.6.0 (August 2025), actively maintained.

Option 5: counterfeiter

counterfeiter generates self-contained fake implementations with no runtime reflection.

Pros Cons
No runtime synchronization Smallest community (1.1k stars)
Actively maintained Different generator and API
Generated fakes are straightforward Introduces another testing pattern

counterfeiter: 1.1k stars, v6.12.2 (April 2026), actively maintained.

Option 6: minimock

minimock generates type-safe mocks with built-in call count verification and timeout-based assertion.

Pros Cons
Type-safe generated code Smaller community than gomock/testify
Built-in call count verification Different API from testify
Timeout-based verification for async tests

minimock: 690 stars. See comparison.

Interface complexity

controller/artifact mocks 11 interfaces totaling 93 methods. Any replacement must cover all of them.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions