How do I report a test that's not run - add `ExplicitTestNodeStateProperty`

[bradwilson]

In the Microsoft.Testing.Platform object models under Microsoft.Testing.Platform.Extensions.Messages, I see messages for passed, failed, and skipped tests.

In xUnit.net v3, we also have a concept of a test that wasn't run because it's only ever run when explicitly asked to. It's philosophically closest to a skipped test, but it's not exactly the same. Is there a plan for anything like this in the new object model? I'm leaning towards not reporting anything at the moment, because dotnet test does not need to say anything. However, I am wondering whether there's a way to highlight such tests in Test Explorer today (and if not, if it's a feature under considering). By not reporting it, it will always just stay in the tree as "not run" which is the most appropriate UX if there's no concept of "won't be run unless you explicitly ask it to be run".

[bradwilson]

Note: I did try using CancelledTestNodeStateProperty but that gets surfaced as a test failure, which this clearly is not. A concern I have is that if the test is reported as "in progress" but then nothing ever happens (no results are reported), I'm not 100% sure how the system will handle this (both on the dotnet test side and in the various UI-based runners like Test Explorer).

[MarcoRossignoli]

A concern I have is that if the test is reported as "in progress" but then nothing ever happens

You should not report this state for a test that is not "acting".
The object model is open to new states so we can think to have something like StagedTestNodeStateProperty or PausedTestNodeStateProperty , @Evangelink do you have some better naming?

[nohwnd]

I think the same concept exists in Nunit, https://docs.nunit.org/articles/nunit/writing-tests/attributes/explicit.html. In my view this should map to a skipped test + reason in TestExplorer. This way it is not making the categories of tests too complicated, and it is easy for the user to understand what happened. Definitely easier than seeing the blue "not run" icon, which is often symptom of an error to run tests (at least currently).

[bradwilson]

You should not report this state for a test that is not "acting".

Because of extensibility, from an execution pipeline perspective we may not know that a test isn't being run until after we find out that it's not being run; which is to say, we may believe we are starting a test that we then discover won't actually be running.

It's philosophically different than skipped. An explicit test can be run if you ask it to, but unless you explicitly ask for it (or ask for all explicit tests to be run), it won't. There's no way to force run a test which is skipped.

In my view this should map to a skipped test + reason in TestExplorer

I'm fine reporting it that way, but I do think it might be confusing to try to express both "this skipped test can't be run even if you ask" and "this skipped test can be run if you ask" with a single UI expression. I understand if you'd rather not make that distinction in the UI of Test Explorer, though it would be still nice if there was some additional metadata (if not a different state entirely) that some other test runner may be able to take advantage of.

[MarcoRossignoli]

I would keep current and future IDEs representation and behavior out of the discussion, I think that the platform should be written without specific "caller" implementation in mind, I'm more for custom tailored made state that a caller needs to recognize and use or ignore.

[Evangelink]

I also think it's better to have an explicit model that represents that states even if current IDEs are not able to handle it.

I am still thinking that the ideal would be to somehow have some kind of handshake between a client and the framework to align on supported states and mapping between them. We are still a bit too early on the design of this concept for now.

About this "explicit" state, this seems to me this is some kind of specialized discovery more than a special run state. From the few tests I have done, the current experience is that explicit tests are only run if there are only explicit tests in the list of tests you receive otherwise they are "skipped" (no result returned for them). which seems to confirm my feeling of specialized discovery. @bradwilson do you think this is a good definition of this state?

[nohwnd]

I would keep current and future IDEs representation and behavior out of the discussion, I think that the platform should be written without specific "caller" implementation in mind,

Sorry for the confusion. My remark was not purely about TestExplorer, but about keeping the number of possible results (or possible result "categories") to the minimum, so implementing a client does not require implementing 100 different flavors of skipped.

In the platform a skipped test result would represent a test that:

• was found
• was not ignored
• was not filtered out
• ended with skipped result

It would NOT represent a test that is not runnable even if included in run by a filter. This is detail of the current test frameworks, and can be conveniently represented as some condition + reporting skipped result.

This allows for the platform to continue working as is and distinguish only those steps:

find tests ("discovery" e.g. via reflection)
ignore tests
filter found tests
report tests ("discovery" => reporting to ui)
run tests
report result

In the run tests phase, the particural framework (or user) can represent the skipped flavors as condition + result as it does now:

-  [Explicit] = if in filter run it as normal, and report result
- [Fact (Skip = "reason" ] = always report skipped right away
- [SkippableFact] = run test, and use code to figure out result Skip.IfNot(Environment.IsWindows);
- or any similar flavor e.g. [WindowsOnly], or [ActiveTest]

The result would then be SkippedTestNodeStateProperty and additional metadata that a UI can optionally represent if it knows about them, but that don't make functional difference to them (it is still a skipped test).

Or it would be some flavor of *SkippedTestNodeStateProperty, but there would be Category (or some other metadata), that would make this map to a Skipped test when the client does not know about the details.

---

If the current way that xunit works is that [explicit] tests are not reported at all (ignored), then I would suggest to change that, and report them as skipped.

---

This all also nicely aligns with the run&discover command that cannot use previously reported (discovered) tests.

[bradwilson]

If the current way that xunit works is that [explicit] tests are not reported at all (ignored), then I would suggest to change that, and report them as skipped.

If you mean when I run them in my own UI (aka, dotnet run), then they are reported as "not run", which is a separate category from "skipped".

If you mean when I report them via the Testing Platform, I will report them as skipped based on this conversion, with some standardized language about why they were skipped.

I'm going to open a second issue related to metadata.