Skip to content

Support the pedantic API #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

art049
Copy link
Member

@art049 art049 commented Jun 3, 2025

Fixes #77

We'll merge both after your PR @not-matthias

@art049 art049 force-pushed the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch from 209b49d to 57148a1 Compare June 5, 2025 20:22
Copy link
Member

@adriencaccia adriencaccia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

It would make sense to test the benchmark.pedantic function in tests/test_pytest_plugin_walltime.py

@art049 art049 force-pushed the cod-949-support-pytest-benchmark-marker-attributes branch 2 times, most recently from 84605a0 to 22d4d33 Compare June 6, 2025 09:17
Base automatically changed from cod-949-support-pytest-benchmark-marker-attributes to master June 6, 2025 10:18
@art049
Copy link
Member Author

art049 commented Jun 6, 2025

Yep I'll add a nominal test for both!

@art049 art049 force-pushed the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch from 57148a1 to 9e0797e Compare June 6, 2025 11:36
Copy link

codspeed-hq bot commented Jun 6, 2025

CodSpeed Instrumentation Performance Report

Merging #81 will not alter performance

Comparing cod-955-allow-stateful-benchmark-replaying-setup-teardown-with (96fe457) with master (22d4d33)

Summary

✅ 57 untouched benchmarks

@art049 art049 force-pushed the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch from 9e0797e to 4875846 Compare June 6, 2025 11:44
Copy link

codspeed-hq bot commented Jun 6, 2025

CodSpeed Walltime Performance Report

Merging #81 will degrade performances by 7.67%

Comparing cod-955-allow-stateful-benchmark-replaying-setup-teardown-with (96fe457) with master (22d4d33)

Summary

⚡ 7 improvements
❌ 21 regressions
✅ 29 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
test_iir_filter_process 3.1 µs 3 µs +2.63%
test_iir_filter_set_coefficients[a_coeffs0-b_coeffs0] 842.8 ns 881.7 ns -4.41%
test_make_allpass 5.1 µs 5.3 µs -3.62%
test_make_bandpass 5.3 µs 5.6 µs -4.13%
test_make_highpass 5.4 µs 5.7 µs -5.44%
test_make_lowpass 5.3 µs 5.6 µs -5.03%
test_combination_lists[0-0] 1.9 µs 1.9 µs +2.34%
test_combination_sum[candidates0-8] 10.7 µs 11 µs -3.04%
test_depth_first_search[4] 30.3 µs 30.9 µs -2.07%
test_generate_all_combinations[0-0] 1.3 µs 1.3 µs -2.49%
test_generate_all_combinations[4-2] 7 µs 7.3 µs -3.77%
test_generate_all_combinations[5-4] 9.9 µs 10.2 µs -3.56%
test_generate_all_permutations[sequence0] 158.2 µs 161.7 µs -2.15%
test_generate_sum_of_subsets_soln[nums0-9] 25.5 µs 24.3 µs +4.79%
test_get_valid_pos[pos0-4] 2.8 µs 2.9 µs -3.26%
test_hamilton_cycle[graph0] 28.2 µs 27.6 µs +2.41%
test_match_word_pattern[aba-GraphTreesGraph] 104.8 µs 101.1 µs +3.61%
test_minimax[scores0] 10.4 µs 10.8 µs -4.37%
test_solve_maze[maze0] 22.8 µs 23.9 µs -4.56%
test_solve_power_sum[13-2] 6.9 µs 7 µs -2.71%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@art049 art049 force-pushed the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch from 4875846 to f38528d Compare June 6, 2025 11:58
@art049 art049 force-pushed the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch from f38528d to 96fe457 Compare June 6, 2025 12:19
@art049 art049 merged commit 96fe457 into master Jun 6, 2025
24 checks passed
@art049 art049 deleted the cod-955-allow-stateful-benchmark-replaying-setup-teardown-with branch June 6, 2025 12:25
@flying-sheep
Copy link

Please revert the typing changes. E.g. args: tuple is just wrong.

self._called = False

def __call__(
self, target: Callable[..., T], *args: tuple, **kwargs: dict[str, Any]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this used to be correct, now it makes no sense at all.

what you changed it to is “each positional argument is a tuple, and each keyword argument is a dict”. So this is valid according to your typing:

benchmark(print, (), (1,2), file={})

but this is invalid according to your typing:

benchmark(print, "a", file="out.txt")

flying-sheep added a commit to flying-sheep/pytest-codspeed that referenced this pull request Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support pytest-benchmark's pedantic API
3 participants