Allow running tests by line and column #1604

jakemac53 · 2021-10-15T23:24:21Z

Allows querying tests by line & col using test path query strings
Matches if any frame in the test stack trace matches up (this allows for lots of flexibility in terms of testing styles and parameterized tests).

~~This does not currently work on browsers, but I probably could get it to? Not sure how important that ultimately is.~~

If a stack trace formatter is set up for the remote platform, it will use it to format the trace sent back for groups/tests that are discovered (enabling browser/node support).

cc @DanTup

pkgs/test/README.md

natebosch · 2021-10-15T23:46:54Z

pkgs/test/README.md

+- **full-name**: Requires an match for the name of the test.
+- **line**: Matches any test that originates from this line in the test suite.
+  - Not supported for browser tests.
+- **col**: Matches any test that originates from this column in the test suite.


I'm confused who would use this? Is this a specific column? Like the position of the first t in test(?

Yes - only IDEs would use it most likely

The reason for it is protection against accidentally running tests the user doesn't expect if they have duplicate names:

// Click "Run" next to this test currently runs both tests, which seems odd to the user test('foo', () {}); test('foo', () {});

Although the IDE will need to be sure of the package:test version before trying to use these new features.

Ya and having the col option specifically guards against having two tests on the same line. A rare situation but its trivial to support once we are already supporting filtering by line.

I still don't understand the precise position this matches.

Is it exactly the first t in test? Is it 0 based? Does this match the same frame as the line number, or can the line match one frame and the column match a different frame?

I'll read through the test cases tomorrow to find out more, but I do think we should expand the user facing docs.

It matches the column position of the function invocation in the stack trace. That is the position of the first character of the name of the function (so if your cursor is to the left of it). It is one based, which matches what the IDE reports (or at least VsCode, I assume this is standard?).

I'll read through the test cases tomorrow to find out more, but I do think we should expand the user facing docs.

There is a sentence after these that says "all filters must match for a test to be ran". I could move it to the top?

Ok I added a mini section under this describing the specific line/col semantics, and also moved the note about all filters matching up to the first paragraph.

pkgs/test_core/lib/src/runner.dart

DanTup · 2021-10-18T10:58:05Z

Matches if any frame in the test stack trace matches up (this allows for lots of flexibility in terms of testing styles and parameterized tests).

I presume this is to support things like testWidgets where the actual test call is inside some helper, and the IDE is passing the line number for the testWidgets call? If so, does it filter the call stack by the filename too?

eg. if I pass my_foo_test.dart?line=10&col=10 I would expect it to match my_foo_test.dart on line/col 10 anywhere in the call stack, but not other files that happens to have a 10:10 in the call stack? (for example, if you happened to create a testWidget that was on the same line as the real test call inside testWidget, you wouldn't want all tests to run).

(although, maybe that wouldn't be a big deal if it's encouraged to always pass a name, even when using line/col).

jakemac53 · 2021-10-18T13:25:27Z

I presume this is to support things like testWidgets where the actual test call is inside some helper, and the IDE is passing the line number for the testWidgets call?

Yes exactly, it essentially allows for "parameterized tests". Probably the IDE doesn't have enough info to know about these actual call sites though (the "root_line" etc would be the actual test call still, since its the first frame in the current file).

If so, does it filter the call stack by the filename too?

Yes it must match the test suite path.

DanTup · 2021-10-18T13:28:37Z

Great, thanks! :-)

…m if a formatter is available

jakemac53 · 2021-10-18T15:07:34Z

Added support and a test for browser/node tests, turned out to be pretty easy :)

pkgs/test/test/runner/line_and_col_test.dart

natebosch · 2021-10-19T03:08:46Z

pkgs/test/README.md

+- **full-name**: Requires an match for the name of the test.
+- **line**: Matches any test that originates from this line in the test suite.
+  - Not supported for browser tests.
+- **col**: Matches any test that originates from this column in the test suite.


I still don't understand the precise position this matches.

Is it exactly the first t in test? Is it 0 based? Does this match the same frame as the line number, or can the line match one frame and the column match a different frame?

I'll read through the test cases tomorrow to find out more, but I do think we should expand the user facing docs.

Co-authored-by: Nate Bosch <[email protected]>

natebosch · 2021-10-19T17:44:19Z

pkgs/test_core/lib/src/runner.dart

+          var line = suite.config.line;
+          var col = suite.config.col;
+          var trace = test.trace;
+          if (trace == null && (line != null || col != null)) {


When would we be missing a trace? Should we consider it an error to try to run tests with a line and column when we have no trace available?

I am not sure if we get load tests and things through this api. I will try making it an error and see what happens.

For now I am just throwing a StateError, let me know if you think a different one would be more appropriate.

pkgs/test_core/lib/src/runner.dart

…add test

pkgs/test/CHANGELOG.md

jakemac53 added 3 commits October 15, 2021 16:14

allow running tests by line or column

d48656f

add tests, update changelog, update readme

336e451

add note about browser test support

e5bbd56

jakemac53 requested a review from natebosch October 15, 2021 23:24

google-cla bot added the cla: yes label Oct 15, 2021

natebosch reviewed Oct 15, 2021

View reviewed changes

jakemac53 added 3 commits October 15, 2021 16:55

code review updates

067cfbe

Merge branch 'master' into run-by-line-col

ca9d827

add call to `precompileTestExecutable

1a82532

jakemac53 added 2 commits October 18, 2021 07:54

add browser test support - format stack traces before serializing the…

307cd27

…m if a formatter is available

add tests

8c9c045

update docs

b9c65d6

jakemac53 requested a review from natebosch October 18, 2021 15:49

natebosch reviewed Oct 19, 2021

View reviewed changes

Update copyright date

81f0320

Co-authored-by: Nate Bosch <[email protected]>

natebosch reviewed Oct 19, 2021

View reviewed changes

jakemac53 added 2 commits October 19, 2021 11:16

update readme to clarify semantics of line/col matching, fix bug and …

0f4b7f9

…add test

throw an error if the trace or path aren\'t available

3cce004

natebosch approved these changes Oct 19, 2021

View reviewed changes

pkgs/test/CHANGELOG.md Outdated Show resolved Hide resolved

remove line from changelog

946bf28

jakemac53 merged commit 8fcea42 into master Oct 19, 2021

jakemac53 deleted the run-by-line-col branch October 19, 2021 21:19

felangel mentioned this pull request Oct 23, 2021

[SEVERE] test_api v0.4.6 causes "Timed out waiting for Chrome to connect." #1620

Closed

DanTup mentioned this pull request Nov 9, 2022

Support running multiple tests by line/col (as package:test does) flutter/flutter#114968

Closed

DanTup mentioned this pull request Feb 16, 2023

[flutter_tools] Add support for URI formats like ?line=x for "flutter test" flutter/flutter#119740

Merged

8 tasks

Allow running tests by line and column #1604

Allow running tests by line and column #1604

Uh oh!

Conversation

jakemac53 commented Oct 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakemac53 Oct 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DanTup commented Oct 18, 2021

Uh oh!

jakemac53 commented Oct 18, 2021

Uh oh!

DanTup commented Oct 18, 2021

Uh oh!

jakemac53 commented Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakemac53 commented Oct 15, 2021 •

edited

Loading

jakemac53 Oct 15, 2021 •

edited

Loading

jakemac53 commented Oct 18, 2021 •

edited

Loading