Skip to content

Allow running tests by line and column #1604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Oct 19, 2021
Merged

Allow running tests by line and column #1604

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d48656f
allow running tests by line or column
jakemac53 Oct 15, 2021
336e451
add tests, update changelog, update readme
jakemac53 Oct 15, 2021
e5bbd56
add note about browser test support
jakemac53 Oct 15, 2021
067cfbe
code review updates
jakemac53 Oct 15, 2021
ca9d827
Merge branch 'master' into run-by-line-col
jakemac53 Oct 16, 2021
1a82532
add call to `precompileTestExecutable
jakemac53 Oct 16, 2021
307cd27
add browser test support - format stack traces before serializing the…
jakemac53 Oct 18, 2021
8c9c045
add tests
jakemac53 Oct 18, 2021
b9c65d6
update docs
jakemac53 Oct 18, 2021
81f0320
Update copyright date
jakemac53 Oct 19, 2021
0f4b7f9
update readme to clarify semantics of line/col matching, fix bug and …
jakemac53 Oct 19, 2021
3cce004
throw an error if the trace or path aren\'t available
jakemac53 Oct 19, 2021
946bf28
remove line from changelog
jakemac53 Oct 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions pkgs/test/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
## 1.19.0-dev

* Support query parameters `name` and `full-name` on test paths, which will
apply the filters to only those test suites.
* Support query parameters `name`, `full-name`, `line`, and `col` on test paths,
which will apply the filters to only those test suites.
* All specified filters must match for a test to run.
* Global filters (ie: `--name`) are also still respected and must match.
* The `line` and `col` will match if any frame from the test trace matches
(the test trace is the current stack trace where `test` is invoked).
* If either `line` or `col` are used, stack trace chaining is enabled for
that suite, regardless of the global setting.
* Give a better exception when using `markTestSkipped` outside of a test.

## 1.18.2
Expand Down
32 changes: 28 additions & 4 deletions pkgs/test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,17 +173,41 @@ description (including any group descriptions) match that regular expression
will be run. You can also use the `-N` flag to run tests whose names contain a
plain-text string.

Alternatively, you can filter tests by name only for a specific file/directory
by specifying a query on a path: `dart test "path/to/test.dart?name=test name"`.
That ensures that the name filters applies to one path but not others.

By default, tests are run in the Dart VM, but you can run them in the browser as
well by passing `dart test -p chrome path/to/test.dart`. `test` will take
care of starting the browser and loading the tests, and all the results will be
reported on the command line just like for VM tests. In fact, you can even run
tests on both platforms with a single command: `dart test -p "chrome,vm"
path/to/test.dart`.

### Test Path Queries

Some query parameters are supported on test paths, which allow you to filter the
tests that will run within just those paths. These filters are merged with any
global options that are passed, and all filters must match for a test to be ran.

- **name**: Works the same as `--name` (simple contains check).
- This is the only option that supports more than one entry.
- **full-name**: Requires an exact match for the name of the test.
- **line**: Matches any test that originates from this line in the test suite.
- **col**: Matches any test that originates from this column in the test suite.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused who would use this? Is this a specific column? Like the position of the first t in test(?

Copy link
Contributor Author

@jakemac53 jakemac53 Oct 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - only IDEs would use it most likely

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for it is protection against accidentally running tests the user doesn't expect if they have duplicate names:

// Click "Run" next to this test currently runs both tests, which seems odd to the user
test('foo', () {});

test('foo', () {});

Although the IDE will need to be sure of the package:test version before trying to use these new features.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya and having the col option specifically guards against having two tests on the same line. A rare situation but its trivial to support once we are already supporting filtering by line.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't understand the precise position this matches.

Is it exactly the first t in test? Is it 0 based? Does this match the same frame as the line number, or can the line match one frame and the column match a different frame?

I'll read through the test cases tomorrow to find out more, but I do think we should expand the user facing docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It matches the column position of the function invocation in the stack trace. That is the position of the first character of the name of the function (so if your cursor is to the left of it). It is one based, which matches what the IDE reports (or at least VsCode, I assume this is standard?).

I'll read through the test cases tomorrow to find out more, but I do think we should expand the user facing docs.

There is a sentence after these that says "all filters must match for a test to be ran". I could move it to the top?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I added a mini section under this describing the specific line/col semantics, and also moved the note about all filters matching up to the first paragraph.


**Example Usage**: `dart test "path/to/test.dart?line=10&col=2"`

#### Line/Col Matching Semantics

The `line` and `col` filters match against the current stack trace taken from
the invocation to the `test` function, and are considered a match if
**any frame** in the trace meets **all** of the following criteria:

* The URI of the frame matches the root test suite uri.
* This means it will not match lines from imported libraries.
* If both `line` and `col` are passed, both must match **the same frame**.
* The specific `line` and `col` to be matched are defined by the tools creating
the stack trace. This generally means they are 1 based and not 0 based, but
this package is not in control of the exact semantics and they may vary based
on platform implementations.

### Sharding Tests
Tests can also be sharded with the `--total-shards` and `--shard-index` arguments,
allowing you to split up your test suites and run them separately. For example,
Expand Down
Loading