Don't tune manually set evals parameter#318
Conversation
carstenbauer
left a comment
carstenbauer
left a comment
There was a problem hiding this comment.
FWIW, LGTM. (But I'm not a maintainer.)
I'm a maintainer but I have only been one for a few days 🤣 |
| - `samples`: The number of samples to take. Execution will end if this many samples have been collected. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.samples = 10000`. | ||
| - `seconds`: The number of seconds budgeted for the benchmarking process. The trial will terminate if this time is exceeded (regardless of `samples`), but at least one sample will always be taken. In practice, actual runtime can overshoot the budget by the duration of a sample. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.seconds = 5`. | ||
| - `evals`: The number of evaluations per sample. For best results, this should be kept consistent between trials. A good guess for this value can be automatically set on a benchmark via `tune!`, but using `tune!` can be less consistent than setting `evals` manually. Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.evals = 1`. | ||
| - `evals`: The number of evaluations per sample. For best results, this should be kept consistent between trials. A good guess for this value can be automatically set on a benchmark via `tune!`, but using `tune!` can be less consistent than setting `evals` manually (which bypasses tuning). Defaults to `BenchmarkTools.DEFAULT_PARAMETERS.evals = 1`. If the function you study mutates its input, it is probably a good idea to set `evals=1` manually. |
There was a problem hiding this comment.
I think a clearer API might be to introduce a special value like evals=nothing to indicate an unset evals.
The text could then something look like this:
evals: The number of evaluations per sample. For best results, this should be kept consistent between trials. If not specified manually, defaults toBenchmarkTools.DEFAULT_PARAMETERS.evals = nothingandtune!is used to automatically set the value to a good guess for the benchmark.Functions which are sensitive to additional un-benchmarked evaluations (e.g. mutating functions), can set
evalsmanually to ensure no automatic tuning is performed.
There was a problem hiding this comment.
I agree that it would be clearer, and that is what @vchuravy suggested at first. There are two reasons I chose the evals_set::Bool implementation instead:
- It's a bit convoluted to remember that
evals=nothingreally meansevals=1 - Much of the code relies on a numeric value for
evals, so my choice was a way of fixing these numerous bugs without checking every timeif isnothing(evals)
|
feel free to weigh in here |
5 participants
This PR add a boolean
evals_setfield to benchmark parameters for manually set evals. Ifevals_set = true, tuning does nothing.Fixes #24 and related issues