Skip to content

Commit 0159b88

Browse files
AndreyAkinshinAndreyAkinshin
committed
[docs] Update vstest
1 parent 2990951 commit 0159b88

File tree

1 file changed

+97
-31
lines changed

1 file changed

+97
-31
lines changed

‎docs/articles/features/vstest.md

Lines changed: 97 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,108 @@
11
---
2-
uid: docs.baselines
2+
uid: docs.vstest
33
name: Running with VSTest
44
---
55

66
# Running with VSTest
7-
BenchmarkDotNet has support for discovering and executing benchmarks through VSTest. This provides an alternative user experience to running benchmarks with the CLI and may be preferable for those who like their IDE's VSTest integrations that they may have used when running unit tests.
7+
8+
BenchmarkDotNet supports discovering and executing benchmarks through VSTest.
9+
This provides an alternative user experience to running benchmarks with the CLI
10+
and may be preferable for those who like their IDE's VSTest integrations that they may have used when running unit tests.
811

912
Below is an example of running some benchmarks from the BenchmarkDotNet samples project in Visual Studio's Test Explorer.
1013

1114
![](../../images/vs-testexplorer-demo.png)
1215

1316
## About VSTest
1417

15-
VSTest is one of the most popular test platforms in use in the .NET ecosystem, with test frameworks such as MSTest, xUnit, and NUnit providing support for it. Many IDEs, including Visual Studio and Rider, provide UIs for running tests through VSTest which some users may find more accessible than running them through the command line.
18+
VSTest is one of the most popular test platforms in use in the .NET ecosystem,
19+
with test frameworks such as MSTest, xUnit, and NUnit providing support for it.
20+
Many IDEs, including Visual Studio and Rider, provide UIs for running tests through VSTest
21+
which some users may find more accessible than running them through the command line.
1622

17-
It may seem counterintuitive to run performance tests on a platform that is designed for unit tests that expect a boolean outcome of "Passed" or "Failed", however VSTest provides good value as a protocol for discovering and executing tests. In addition, we can still make use of this boolean output to indicate if the benchmark had validation errors that caused them to fail to run.
23+
It may seem counterintuitive to run performance tests on a platform
24+
that is designed for unit tests that expect a boolean outcome of "Passed" or "Failed".
25+
However, VSTest provides good value as a protocol for discovering and executing tests.
26+
In addition, we can still make use of this boolean output to indicate
27+
if the benchmark had validation errors that caused it to fail to run.
1828

1929
## Caveats and things to know
20-
- The VSTest adapter will not call your application's entry point.
21-
- If you use the entry point to customize how your benchmarks are run, you will need to do this through other means such as an assembly-level `IConfigSource`.
22-
- For more about this, please read: [Setting a default configuration](#setting-a-default-configuration).
23-
- The benchmark measurements may be affected by the VSTest host and your IDE
24-
- If you want to have more accurate performance results, it is recommended to run benchmarks through the CLI instead without other processes on the machine impacting performance.
25-
- This does not mean that the measurements are useless though, it will still be able to provide useful measurements during development when comparing different approaches.
26-
- The test adapter will not display or execute benchmarks if optimizations are disabled.
27-
- Please ensure you are compiling in Release mode or with `Optimize` set to true.
28-
- Using an `InProcess` toolchain will let you run your benchmarks with optimizations disabled and will let you attach the debugger as well.
29-
- The test adapter will generate an entry point for you automatically
30-
- The generated entry point will pass the command line arguments and the current assembly into `BenchmarkSwitcher`, so you can still use it in your CLI as well as in VSTest.
31-
- This means you can delete your entry point and only need to define your benchmarks.
32-
- If you want to use a custom entry point, you can still do so by setting `GenerateProgramFile` to `false` in your project file.
33-
34-
## How to use it
35-
36-
You need to install two packages into your benchmark project:
3730

38-
- `BenchmarkDotNet.TestAdapter`: Implements the VSTest protocol for BenchmarkDotNet
39-
- `Microsoft.NET.Test.Sdk`: Includes all the pieces needed for the VSTest host to run and load the VSTest adapter.
40-
41-
As mentioned in the caveats section, `BenchmarkDotNet.TestAdapter` will generate an entry point for you automatically, so if you have an entry point already you will either need to delete it or set `GenerateProgramFile` to `false` in your project file to continue using your existing one.
31+
* **The benchmark measurements may be affected by the VSTest host and your IDE!**
32+
If you want to have accurate measurements,
33+
it is recommended to run benchmarks through the CLI without other processes on the machine impacting performance.
34+
This does not mean that the measurements are useless though,
35+
it will still be able to provide useful measurements during development when comparing different approaches.
36+
* **The test adapter will not display or execute benchmarks if optimizations are disabled.**
37+
Please ensure you are compiling in Release mode or with `Optimize` set to true.
38+
Using an `InProcess` toolchain will let you run your benchmarks with optimizations disabled
39+
and will let you attach the debugger as well.
40+
* **The VSTest adapter will not call your application's entry point.**
41+
If you use the entry point to customize how your benchmarks are run,
42+
you will need to do this through other means such as an assembly-level `IConfigSource`,
43+
as shown [here](#setting-a-default-configuration).
44+
* **The test adapter will generate an entry point for you automatically.**
45+
The generated entry point will pass the command line arguments
46+
and the current assembly into `BenchmarkSwitcher`,
47+
so you can still use it in your CLI as well as in VSTest.
48+
This means you can delete your entry point and only need to define your benchmarks.
49+
If you want to use a custom entry point, you can still do so by setting `GenerateProgramFile` to `false` in your project file.
50+
51+
## Getting started
52+
53+
* **Step 1.** Install the NuGet packages.
54+
You need to install two packages into your benchmark project:
55+
* `BenchmarkDotNet.TestAdapter`: Implements the VSTest protocol for BenchmarkDotNet
56+
* `Microsoft.NET.Test.Sdk`: Includes all the pieces needed for the VSTest host to run and load the VSTest adapter.
57+
* **Step 2.** Make sure that the entry point is configured correctly.
58+
As mentioned in the caveats section, `BenchmarkDotNet.TestAdapter` will generate an entry point for you automatically.
59+
So, if you have an entry point already,
60+
you will either need to delete it or set `GenerateProgramFile` to `false` in your project file to continue using your existing one.
61+
Here is an example of a `.csproj` file based on the default Console Application template:
62+
63+
```xml
64+
<Project Sdk="Microsoft.NET.Sdk">
65+
66+
<PropertyGroup>
67+
<OutputType>Exe</OutputType>
68+
<TargetFramework>net8.0</TargetFramework>
69+
<ImplicitUsings>enable</ImplicitUsings>
70+
<Nullable>enable</Nullable>
71+
<!-- Disable entry point generation as this project has it's own entry point -->
72+
<GenerateProgramFile>false</GenerateProgramFile>
73+
</PropertyGroup>
74+
75+
<ItemGroup>
76+
<PackageReference Include="BenchmarkDotNet.TestAdapter" Version="0.13.12" />
77+
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
78+
</ItemGroup>
79+
80+
</Project>
81+
```
4282

43-
After doing this, you can set your build configuration to `Release`, run a build, and you should be able to see the benchmarks in your IDE's VSTest integration.
83+
* **Step 3.** Make sure that your IDE supports VSTest integration.
84+
In Visual Studio, everything works out of the box.
85+
In Rider/R# 2023.3, the VSTest integration should be activated:
86+
* Go to the "Unit Testing" settings page.
87+
* Rider: Settings -> Build, Execution, Deployment -> Unit Testing -> VSTest
88+
* R#: Extensions -> ReSharper -> Options -> Tools -> Unit Testing -> Test Frameworks -> VSTest
89+
* Make sure that the "Enable VSTest adapter support" checkbox is checked.
90+
* Add your project name to the "Projects with Unit Tests" list (or `*` to enable it for all projects).
91+
In future versions of Rider, this should be enabled by default.
92+
* **Step 4.** Switch to the `Release` configuration.
93+
As mentioned above, the TestAdapter is not able to discover and run benchmarks with optimizations disabled (by design).
94+
* **Step 5.** Build the project.
95+
In order to discover the benchmarks, the VSTest adapter needs to be able to find the assembly.
96+
Once you build the project, you should observe the discovered benchmarks in your IDE's Unit Test Explorer.
97+
98+
If you correctly performed all the steps above, you should be able to run your benchmarks in your IDE using embedded unit testing features.
99+
If this doesn't work for you, don't hesitate to file [a new GitHub issue](https://github.com/dotnet/BenchmarkDotNet/issues/new).
44100

45101
## Setting a default configuration
46102

47-
Previously, it was common for the default configuration to be defined inside the entry point. Since the entry point is not used when running benchmarks through VSTest, the default configuration must be specified using a `Config` attribute instead that is set on the assembly.
103+
Previously, it was common for the default configuration to be defined inside the entry point.
104+
Since the entry point is not used when running benchmarks through VSTest,
105+
the default configuration must be specified using a `Config` attribute that is set on the assembly instead.
48106

49107
First, create a class that extends `ManualConfig` or `IConfig` which sets the default configuration you want:
50108

@@ -54,8 +112,8 @@ class MyDefaultConfig : ManualConfig
54112
public MyDefaultConfig()
55113
{
56114
AddJob(Job.Dry);
57-
AddLogger(Loggers.ConsoleLogger.Default);
58-
AddValidator(JitOptimizationsValidator.DontFailOnError);
115+
AddLogger(Loggers.ConsoleLogger.Default);
116+
AddValidator(JitOptimizationsValidator.DontFailOnError);
59117
}
60118
}
61119
```
@@ -69,6 +127,14 @@ Then, set an assembly attribute with the following.
69127
By convention, assembly attributes are usually defined inside `AssemblyInfo.cs` in a directory called `Properties`.
70128

71129
## Viewing the results
72-
The full output from BenchmarkDotNet that you would have been used to seeing in the past will be sent to the "Tests" output of your IDE. Use this view if you want to see the tabular view that compares multiple benchmarks with each other, or if you want to see the results for each individual iteration.
73130

74-
One more place where you can view the results is in each individual test's output messages. In Visual Studio this can be viewed by clicking on the test in the Test Explorer after running it, and looking at the Test Detail Summary. Since this only displays statistics for a single benchmark case, it does not show the tabulated view that compares multiple benchmark cases, but instead displays a histogram and various other useful statistics. Not all IDEs support displaying these output messages, so you may only be able to view the results using the "Tests" output.
131+
The full output from BenchmarkDotNet that you would have been used to seeing in the past will be sent to the "Tests" output of your IDE.
132+
Use this view if you want to see the tabular view that compares multiple benchmarks with each other or
133+
if you want to see the results for each individual iteration.
134+
135+
One more place where you can view the results is in each individual test's output messages.
136+
In Visual Studio, this can be viewed by clicking on the test in the Test Explorer after running it and looking at the Test Detail Summary.
137+
Since this only displays statistics for a single benchmark case,
138+
it does not show the tabulated view that compares multiple benchmark cases.
139+
Instead, it displays a histogram and various other useful statistics.
140+
Not all IDEs support displaying these output messages, so you may only be able to view the results using the "Tests" output.

0 commit comments

Comments
 (0)