Skip to content

Make it possible to uniquely identify test cases #1328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
natebosch opened this issue Aug 25, 2020 · 1 comment
Open

Make it possible to uniquely identify test cases #1328

natebosch opened this issue Aug 25, 2020 · 1 comment

Comments

@natebosch
Copy link
Member

There are a few use cases that are easier to tackle when a test case can be uniquely identified.

Some options to consider:

  • If we can give a unique ID to a suite, we could introduce a restriction that test names are unique within a test suite (generally a single test file). Then a suite ID and test name would be unique. Should two tests with the same name be an error? #661 This would be breaking but is something we could feasibly roll out.
  • We could use a deterministic ordering (if it isn't already, it may be for sharding purposes) for test suites to load and number test cases as they are declared.

This is going to be challenging to take full advantage of. Since tests are declared by executing Dart code there is no static and consistent set of tests. There are any number of ways a tests could introduce non-determinism in the order that test cases are declared, or even in the number and names of tests. If we consider any pattern of interaction that involves separate invocations of the test runner we might want to build in some safety from the start - for instance we could create a hash of some information when we assign names to tests, and then require that it is passed to us and still matches when we run a named test. (We could use a similar concept when sharding, each shard should ideally arrive at the same hash so that we are sure there is a consistent view of the tests and that cases aren't being dropped due to non-determinism).

cc @jiahaog @DanTup

@DanTup
Copy link
Contributor

DanTup commented Aug 25, 2020

Maybe dupe of #837?

In VS Code I worked around the original issue by just matching things based on name. In practice, it works great - with some minor niggles like if you have multiple tests with the same name, we'll run them all when you try to run one. We use the analysis servers Outline data to get tests names so create the CodeLens links though, so I'm not sure unique IDs would solve this since the editor wouldn't have them up-front (though perhaps passing an index number in this case might work - "run the nth test called x").

There are two other issues I have but I don't know if IDs are the solution (perhaps improving the --name param is):

  1. Can't filter on exact group names (we use regex with a leading ^ and trailing $ normally to avoid catching too many tests, but we have to leave the $ off the end for groups, but that can result in matching more than just the intended group).
  2. Can't filter on tests that have variables/etc. in the names (the analysis server gives us the exact string, including the variable, but package:test only matches on the resolved string passed to the test function).

natebosch added a commit that referenced this issue Sep 24, 2020
Towards #1310, #1328

Add `directRunTest` to configure a reporter and run tests directly in
the same isolate.

Add `enumerateTestCases` to collect test names without running them, and
`directRunSingleTest` to run a specific test by its full name. These
APIs ensure the uniqueness of test names to avoid ambiguity. This
restriction may be spread to tests run through the normal test runner as
well.

- Add `fullTestName` option on `Declarer`. When used, only the test (or
  tests if uniqueness is not checked separately) will be considered as a
  test case.
- Add `directRunTest`, `enumerateTestCases`, and `directRunSingleTest`
  APIs. These are kept under a `lib/src/` import for now, and any other
  package that uses these APIs should pin to a specific version of
  `package:test_core`. The details of these APIs might change without a
  major version bump.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants