Produce a warning if multiple tests run with the exact same group/test name

[DanTup]

There's some functionality in Dart-Code that tries to handle this, but it's a rather clunky and I think could be better dealt with here. (I'm not certain it's a common issue, but it did come up for Dart-Code at least once).

Consider a user duplicated a test, but forgot to update the name:

main() {
group('foo', () {
test('bar', () {});
test('bar', () {});
});

Since IDEs can only run tests by name, it can be confusing to click Run next to one test and find yourself debugging code that originated from the other.

Unless there's a good reason to support multiple tests with the same name, it would be nice if the test package could warn when this happens (or even just refuse to run). If it's opt-in with a flag, that would be fine too (the IDE could pass it in the case where it expects exactly one test to run - eg. "Run" is clicked on a test, and not a group).

Another possibility could be to detect this before running with a lint or something (although dynamic test names might complicate that a little)? @pq

[jakemac53]

Ya this is uncommon and a bad practice, we probably should just block/warn for it.

I think this would be hard to lint for, and if you run tests in the IDE you don't necessarily see the warnings.

We could add a warning and an option to upgrade that to an error though, which it would possibly make sense for the IDE to use.

[jakemac53]

Running tests by name has other problems too though (especially around dynamic names).

We could also explore different options for that. For instance one idea that might work better for IDEs specifically is the ability to run a test based on the column/line location in the file where the test invocation is? A bit funky to be sure but we should be able to gather that info by inspecting the stack trace. That requires source maps to be working for web tests, so it could be brittle.

[DanTup]

I think this would be hard to lint for, and if you run tests in the IDE you don't necessarily see the warnings.

Not sure I understand - if it was a real lint, it would show up in the IDE in the normal errors list? (this would show the user issue without needing to run the test, but I think also be less reliable).

Running tests by name has other problems too though (especially around dynamic names).

Yeah, this is another place I have some hacks - for example we try to discover tests using the analyzer outline information (so we can populate the test tree when you open a file without having run it), and currently just skip over any test names with a $ in. It would be nice to improve this area, although I don't have any great ideas.

For instance one idea that might work better for IDEs specifically is the ability to run a test based on the column/line location in the file where the test invocation is?

Yeah, this could work. We'd to make sure as the user types in the file we keep the location updated (for ex. they may modify the file then click Run without saving). Currently we only use the analysis server outline information for the initial discovery, but perhaps we could use it to maintain a live set of tests instead (this could fix one of the outstanding niggles where if you rename a test, the old name stays around until you re-run the whole file since we don't otherwise know it's not there anymore).

[jakemac53]

Not sure I understand - if it was a real lint, it would show up in the IDE in the normal errors list? (this would show the user issue without needing to run the test, but I think also be less reliable).

Sorry what I meant is if we log warnings on the command line, users won't see them since they don't see command line output (and actually we probably wouldn't even log the warning in the json reporter?).

[DanTup]

Sorry what I meant is if we log warnings on the command line, users won't see them since they don't see command line output (and actually we probably wouldn't even log the warning in the json reporter?).

Ah, gotcha. That could be a reason to just reject it and fail with an error (to stderr) instead.

Although @jacob314 suggested a lint may be better (although I guess that could only detect constant test names) and @natebosch may have input from Flutter test perspective.

I'm not sure this is a common issue, though it came up in some discussions about trying to make it easier to match up tests from the source code (eg. using analysis outlines) versus the test runner. Right now it's quite easy to make what's shown in the VS Code test runner not really match what's in the editor and we sometimes have to guess which node in the tree relates to which result (eg. in the case of multiple tests with the same name, we then start looking at line numbers to find the "closest" one - to allow for edits to the file). With some restrictions (eg. don't use tests with the same name or non-constant names) some of that logic could perhaps be simplified.

[jakemac53]

Ah, gotcha. That could be a reason to just reject it and fail with an error (to stderr) instead.

This is what I would prefer in an ideal world, but I think it is probably pretty breaking for folks and maybe not worth doing a breaking release of the package? Although breaking releases of package:test are actually possible, so maybe we could do it.

With some restrictions (eg. don't use tests with the same name or non-constant names) some of that logic could perhaps be simplified.

Note that non-constant names do have real use cases so I wouldn't want to block this. They are problematic for the IDE but useful.

[jacob314]

It would be great if we could have lints and runtime errors that enforce that the tests for a file are exactly the tests that static analysis believes are run.

With that in mind, would be great to see the following static lints and runtime checks which I believe could together ensure that the list of tests the outline detects match the list of tests in the app.

1. Make non-constant test names a lint error
2. Make duplicate test names and duplicate group names a lint error as well as a runtime error in the test framework.
3. Enforce that test group and test block definitions always run if they are defined in a file with a lint and perhaps a runtime check (would require passing in the expected list of tests determined by static analysis)

Examples:

main() {
  group('foo', () {
    test('bar', () {}); // Lint error: duplicate test name
    test('bar', () {});
    test('$baz', () {}); // Lint error:  non constant test name
  });
}

main() {
  group('foo', () {
    test('bar', () {});
  });
  group('baz', () {
    test('bar', () {}); // Ok as bar tests are in separate groups?
  });
}

main() {
  group('foo', () {
    test('bar', () {});
  });
  group('foo', () { // Lint error: duplicate group name.
    test('baz', () {}); 
  });
}

main() {
  if (foo == bar) { // Lint error: use `skip` if a test or group should only sometimes run. Runtime error if test 'foo' is not run.
    group('foo', () {
      test('bar', () {});
    });
  }
}

[DanTup]

// Ok as bar tests are in separate groups?

Yep, tests with overlapping in groups should be fine (the IDE considers the test name to be fully qualified including all parent groups and the test name concatenated with a space).

I think (optional) lints for the above would be neat. It would make it easier to convey to users what rules they should follow to get the best experience in the editor and it may also make it more reasonable for us to drop results out of the runner that we think (from static analysis) no longer exist in the file (so rename tests don't hang around until you run the entire file).

[jakemac53]

• Make non-constant test names a lint error

I am fine with this as an optional lint, but not a default one. There are a lot of situations where it is valid to build up test names today, it is effectively the only way to write parameterized tests in Dart.

[jakemac53]

main() {
  group('foo', () {
    test('bar', () {});
  });
  group('baz', () {
    test('bar', () {}); // Ok as bar tests are in separate groups?
  });
}

Yes this is fine

main() {
  group('foo', () {
    test('bar', () {});
  });
  group('foo', () { // Lint error: duplicate group name.
    test('baz', () {}); 
  });
}

In theory this might be ok but it seems reasonable enough to block. It would lead to weird behavior anyways (you would end up running both groups).

[jakemac53]

Another case that gets interesting:

main() {
  group('foo', () {
    test('bar', () {});
  });
  test('foobaz', () {});
}

I believe that running the foo group today would also end up running the foobaz test.

This might be rare enough that we don't have to care about it.

[DanTup]

I think #908 discusses that issue. We have to use an open-ended regex for groups today, but it we could make it closed (eg. run things in groups matching ^foo$) I think that might solve that.

[jacob314]

• Make non-constant test names a lint error

I am fine with this as an optional lint, but not a default one. There are a lot of situations where it is valid to build up test names today, it is effectively the only way to write parameterized tests in Dart.

Can you give some examples of this? I'd only be interested in adding a lint for it if it was in the default set of lints users were opted into. Our IDE support will be lousy if you author test names like this.

[jakemac53]

I think #908 discusses that issue. We have to use an open-ended regex for groups today, but it we could make it closed (eg. run things in groups matching ^foo$) I think that might solve that.

Ya adding support for running a group by name should be doable I would think, and would probably solve the problem.

Can you give some examples of this? I'd only be interested in adding a lint for it if it was in the default set of lints users were opted into. Our IDE support will be lousy if you author test names like this.

https://github.com/dart-lang/source_gen/blob/master/source_gen/test/generator_for_annotation_test.dart#L14

[jakemac53]

It seems to me like what is really needed here is some sort of interactive mode for the test runner. Something that you can launch and then talk to via a JSON protocol (or some other protocol).

You would be able to ask it for instance what are all the available tests, and what are their IDs. It would also return to you the line/column information for that test. Then you would be able to ask it to run specific tests by ID etc. This could also be used as the basis for a general interactive test runner UI.

I think it gets weird still when you start talking about the IDE experience and what happens when somebody is making edits to a file that they haven't saved yet though. But that is probably weird today as well? Maybe we could actually make that situation better with this model and have it use the unsaved content of the file for the test or something.