|
| 1 | +# Jobs |
| 2 | + |
| 3 | +Basically, *job* describes how to run your benchmark. Practically, it's a set of characteristics which can be specified. You can set one or several jobs for your benchmarks. |
| 4 | + |
| 5 | +## Characteristics |
| 6 | + |
| 7 | +There are several categories of characteristics which you can specify. Let's consider each category in detail. |
| 8 | + |
| 9 | +### Id |
| 10 | +It's a single string characteristics. It allows to name your job. This name will be used in logs and a part of a folder name with generated files for this job. `Id` doesn't affect benchmark results, but it can be useful for diagnostics. If you don't specify `Id`, random value will be chosed based on other characteristics |
| 11 | + |
| 12 | +### Env |
| 13 | +`Env` specifies an environment of the job. You can specify the following characteristics: |
| 14 | + |
| 15 | +* `Platform`: `x86` or `x64` |
| 16 | +* `Runtime`: |
| 17 | + * `Clr`: Full .NET Framework (available only on Windows) |
| 18 | + * `Core`: CoreCLR (x-plat) |
| 19 | + * `Mono`: Mono (x-plat) |
| 20 | +* `Jit`: |
| 21 | + * `LegacyJit` (available only for `Runtime.Clr`) |
| 22 | + * `RyuJit` (avaiable only for `Runtime.Clr` and `Runtime.Core`) |
| 23 | + * `Llvm` (avaiable only for `Runtime.Mono`) |
| 24 | +* `Affinity`: [Affinity](https://msdn.microsoft.com/library/system.diagnostics.process.processoraffinity.aspx) of a benchmark process |
| 25 | +* `GcMode`: settings of Garbage Collector |
| 26 | + * `Server`: `true` (Server mode) or `false` (Workstation mode) |
| 27 | + * `Concurrent`: `true` (Concurrent mode) or `false` (NonConcurrent mode) |
| 28 | + * `CpuGroups`: Specifies whether garbage collection supports multiple CPU groups |
| 29 | + * `Force`: Specifies whether the BenchmarkDotNet's benchmark runner forces full garbage collection after each benchmark invocation |
| 30 | + * `AllowVeryLargeObjects`: On 64-bit platforms, enables arrays that are greater than 2 gigabytes (GB) in total size |
| 31 | + |
| 32 | +BenchmarkDotNet will use host process environment characteristics for non specified values. |
| 33 | + |
| 34 | +### Run |
| 35 | +In this category, you can specifiy how to benchmark each method. |
| 36 | + |
| 37 | +* `RunStrategy`: |
| 38 | + * `Throughput`: default strategy which allows to get good precision level |
| 39 | + * `ColdStart`: should be used only for measuring cold start of the application or testing purpose |
| 40 | +* `LaunchCount`: how many times we should launch process with target benchmark |
| 41 | +* `WarmupCount`: how many warmup iterations should be performed |
| 42 | +* `TargetCount`: how many target iterations should be performed |
| 43 | +* `IterationTime`: desired time of a single iteration |
| 44 | +* `InvocationCount`: count of invocation in a single iteration (if specified, `IterationTime` will be ignored) |
| 45 | + |
| 46 | +Usually, you shouldn't specify such characteristics like `LaunchCount`, `WarmupCount`, `TargetCount`, or `IterationTime` because BenchmarkDotNet has a smart algorithm to choose these values automatically based on recieved measurements. You can specify it for testing purposes or when you are damn sure that you know perfect characteristics for your benchmark (when you set `TargetCount` = `20` you should unserstand why `20` is a good value for your case). |
| 47 | + |
| 48 | +### Accuracy |
| 49 | +If you want to change the accuracy level, you should use the following characteristics instead of manual of values of `WarmupCount`, `TargetCount`, and so on. |
| 50 | + |
| 51 | +* `MaxStdErrRelative`: Maximum relative standard error (`StandardError`/`Mean`) which you want to achive. |
| 52 | +* `MinIterationTime`: Minimum time of a single iteration. Unlike `Run.IterationTime`, this characteristic specify only the lower limit. In case of need, BenchmarkDotNet can increase this value. |
| 53 | +* `MinInvokeCount`: Minimum about of target method invocation. Default value if `4` but you can decrease this value for cases when single invocations takes a lot of time. |
| 54 | +* `EvaluateOverhead`: if you benchmark method takes nanoseconds, BenchmarkDotNet overhead can significantly affect measurements. If this characterics is enable, the overhead will be evaluated and substracted from the result measurements. Default value is `true`. |
| 55 | +* `RemoveOutliers`: sometimes you could have outliers in your measurements. Usually it's *unexpected* ourliers which arised because of other processes activities. If this characteristics is enable, all outliers will be removed from the result measurements. However, some of benchmarks have *expected* outliers. In these situation, you expect that some of invocation can produce ourliers measurements (e.g. in case of network acitivities, cache operations, and so on). If you want to see result statistics with these outliers, you should disable this characteristic. Default value is `true`. |
| 56 | + |
| 57 | +### Infra |
| 58 | +Usually, you shouldn't specify any characteristics from this section, it can be used for advanced cases only. |
| 59 | + |
| 60 | +* `Toolchain`: a toolchain which generate source code for target benchmark methods, build it, and execute it. BenchmarkDotNet has own toolchains for CoreCLR projects and classic projects (the last one is `RoslynToolchain`, you can find it in the [BenchmarkDotNet.Toolchains.Roslyn](https://www.nuget.org/packages/BenchmarkDotNet.Toolchains.Roslyn/) NuGet package). If you want, you can define own toolchain. |
| 61 | +* `Clock`: a clock which will be used for measurements. BenchmarkDotNet automatically choose the best available clock source, but you can specify own clock source. |
| 62 | +* `Engine`: a measurement engine which performs all the measurement magic. If you don't trust BenchmarkDotNet, you can define own engine and implement all the measurement stages manually. |
| 63 | + |
| 64 | +## Usage |
| 65 | + |
| 66 | +There are several ways to specify a job. |
| 67 | + |
| 68 | +### Object style |
| 69 | + |
| 70 | +You can create own jobs directly from the source code via a cusom config: |
| 71 | + |
| 72 | +```cs |
| 73 | +[Config(typeof(Config))] |
| 74 | +public class MyBenchmarks |
| 75 | +{ |
| 76 | + private class Config : ManualConfig |
| 77 | + { |
| 78 | + public Config() |
| 79 | + { |
| 80 | + Add(Job.Default. |
| 81 | + With(Platform.X64). |
| 82 | + With(Jit.RyuJit). |
| 83 | + With(Runtime.Core). |
| 84 | + WithLaunchCount(5). |
| 85 | + WithIterationTime(TimeInterval.Millisecond * 200). |
| 86 | + WithMaxStdErrRelative(0.01). |
| 87 | + WithId("MySuperJob") |
| 88 | + ); |
| 89 | + } |
| 90 | + } |
| 91 | + // Benchmarks |
| 92 | +} |
| 93 | +``` |
| 94 | + |
| 95 | +Basically, it's a good idea to start with one of predefined jobs (e.g. `Job.Default` or `Job.Dry`) and modify it with help of `With*` methods. |
| 96 | + |
| 97 | +### Attribute style |
| 98 | + |
| 99 | +You can also add new jobs via attributes. Examples: |
| 100 | + |
| 101 | +```cs |
| 102 | +[DryJob] |
| 103 | +[ClrJob, CoreJob, MonoJob] |
| 104 | +[LegacyJitX86Job, LegacyJitX64, RyuJitX64Job] |
| 105 | +[SimpleJob(RunStrategy.ColdStart, launchCount: 1, warmupCount: 5, targetCount: 5, id: "FastAndDirtyJob")] |
| 106 | +public class MyBenchmarkClass |
| 107 | +``` |
| 108 | + |
| 109 | +#### Custom attributes |
| 110 | + |
| 111 | +You can also create own custom attribute with your favorite set of jobs. Example: |
| 112 | + |
| 113 | +```cs |
| 114 | +[AttributeUsage(AttributeTargets.Class | AttributeTargets.Assembly)] |
| 115 | +public class MySuperJobAttribute : Attribute, IConfigSource |
| 116 | +{ |
| 117 | + protected MySuperJobAttribute() |
| 118 | + { |
| 119 | + var job = Job.Default.With(Runtime.Core).WithId("MySuperJob"); |
| 120 | + Config = ManualConfig.CreateEmpty().With(job); |
| 121 | + } |
| 122 | + |
| 123 | + public IConfig Config { get; } |
| 124 | +} |
| 125 | + |
| 126 | +[MySuperJob] |
| 127 | +public class MyBenchmarks |
| 128 | +``` |
0 commit comments