-
Notifications
You must be signed in to change notification settings - Fork 218
Make it possible to uniquely identify test cases #1328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Maybe dupe of #837? In VS Code I worked around the original issue by just matching things based on name. In practice, it works great - with some minor niggles like if you have multiple tests with the same name, we'll run them all when you try to run one. We use the analysis servers Outline data to get tests names so create the CodeLens links though, so I'm not sure unique IDs would solve this since the editor wouldn't have them up-front (though perhaps passing an index number in this case might work - "run the nth test called x"). There are two other issues I have but I don't know if IDs are the solution (perhaps improving the
|
Towards #1310, #1328 Add `directRunTest` to configure a reporter and run tests directly in the same isolate. Add `enumerateTestCases` to collect test names without running them, and `directRunSingleTest` to run a specific test by its full name. These APIs ensure the uniqueness of test names to avoid ambiguity. This restriction may be spread to tests run through the normal test runner as well. - Add `fullTestName` option on `Declarer`. When used, only the test (or tests if uniqueness is not checked separately) will be considered as a test case. - Add `directRunTest`, `enumerateTestCases`, and `directRunSingleTest` APIs. These are kept under a `lib/src/` import for now, and any other package that uses these APIs should pin to a specific version of `package:test_core`. The details of these APIs might change without a major version bump.
There are a few use cases that are easier to tackle when a test case can be uniquely identified.
Some options to consider:
This is going to be challenging to take full advantage of. Since tests are declared by executing Dart code there is no static and consistent set of tests. There are any number of ways a tests could introduce non-determinism in the order that test cases are declared, or even in the number and names of tests. If we consider any pattern of interaction that involves separate invocations of the test runner we might want to build in some safety from the start - for instance we could create a hash of some information when we assign names to tests, and then require that it is passed to us and still matches when we run a named test. (We could use a similar concept when sharding, each shard should ideally arrive at the same hash so that we are sure there is a consistent view of the tests and that cases aren't being dropped due to non-determinism).
cc @jiahaog @DanTup
The text was updated successfully, but these errors were encountered: