Tests with breakpoint set being skipped when using --pause-after-load

[TimIrwin26]

When attempting to debug unit tests running on either content-shell or Dartium, any group where I set a breakpoint in one of the tests using browser developer tools ends up skipped.

In the second test run, I set a breakpoint in one of the tests during the "pause-after-load" using Chrome developer tools. The tests run fine if "--pause-after-load" is used with no browser breakpoints.

• OSX 10.11.3
• Dart 1.15.0
• test 1.12.12
• Current version of Chrome (or Dartium shipped in Homebrew package w/ Dart 1.15.0)
• Also reproduces on Ubuntu with a similar setup.

pro:~/sandbox/unit_test$ pub run test -p content-shell test/some_test.dart 
00:04 +0: A should fail                                                                            
abc
00:04 +0 -1: A should fail                                                                         
  Expected: <false>
    Actual: <true>

  package:test         expect
  some_test.dart 11:7  main.<fn>.<fn>

00:04 +0 -1: B should pass                                                                         
abc
00:04 +1 -1: C all called                                                                          
[should fail, should pass]
00:04 +2 -1: Some tests failed.          

pro:~/sandbox/unit_test$ pub run test -p content-shell test/some_test.dart --pause-after-load
00:04 +0: loading test/some_test.dart                                                              

Observatory URL: http://127.0.0.1:51847
Remote debugger URL: http://localhost:51837/devtools/inspector.html?ws=localhost:51837/devtools/page/7341D459-82CC-47A9-9C3C-6A77CA1CF37D
The test runner is paused. Open the remote debugger or the Observatory and set breakpoints. Once
you're finished, return to this terminal and press Enter.

00:04 +0: All tests skipped.      

pubspec.yaml.txt
some_test.dart.txt

[TimIrwin26]

If this problem went away by not needing to use --pause-after-load for debugging, that would be wonderful. From a user's perspective it's an awkward hack at best.

[zoechi]

Actually it's only a hack as far as I know. This is because of limitations in the VM and pub which prevented attaching a debugger for code run with pub run. These issues are already fixed (or at least mostly). I guess this will be fixed in test in the not-so-far future as well.

[TimIrwin26]

It's a bit difficult to convince my developers that TDD (or even unit tests) is a good thing when there hasn't been a reasonable debugging story since they moved to "pub run".

[zoechi]

For me the easiest workaround is to create a copy of the xxx_test.html file and replace the x-dart-test with a normal dart script tag and run the test with debug (without pub run). This way I can debug from within the WebStorm IDE like before. If you make the name of the copy not end with ...test_.html like ..._test_.html then it is ignored by pub run test (not sure if this is even necessary).

As mentioned it is a hack and will be fixed, but there were quite some complications and dependencies from what I saw while following the discussions. This is why it take longer than expected.

[nex3]

I can't reproduce this with Dartium 1.14.2, but I can with Dartium 1.15.0. I'm pretty sure it's a bug in Dartium or Observatory—I suggest you re-file this against the sdk.

I'm surprised to hear you say that --pause-after-load feels like a hack. I'm not sure how else you'd support debugging in the test runner.

[TimIrwin26]

Maybe have something that works within the IDE? With --pause-after-load I have to dig through my source and reset breakpoints every run. I don't think I've had to do that in any other environment (going back to Borland C).

[vpigrad]

This is still a problem. Why was this closed?

[TimIrwin26]

nex3 wanted it filed against the SDK team -- dart-lang/sdk#26080. If this ends up ping-ponging between the teams, I'll be even less happy than I am now.

[nex3]

Maybe have something that works within the IDE?

IDE integration is something to bring up with IDE authors. I can only make the test runner as usuable as possible from the command line; how other tools integrate with it is up to them.

With --pause-after-load I have to dig through my source and reset breakpoints every run. I don't think I've had to do that in any other environment (going back to Borland C).

In 0.12.7 we added support for iterative debugging. When you're paused for debugging, there's a console available on the command line that allows you to re-run tests while keeping breakpoints intact.

[TimIrwin26]

And that is documented where? I've found references to it's existence,
but nothing documenting how to use it.

On 3/23/16 3:19 PM, Natalie Weizenbaum wrote:

Maybe have something that works within the IDE?

IDE integration is something to bring up with IDE authors. I can only
make the test runner as usuable as possible from the command line; how
other tools integrate with it is up to them.

With --pause-after-load I have to dig through my source and reset
breakpoints every run. I don't think I've had to do that in any
other environment (going back to Borland C).

In 0.12.7 we added support for iterative debugging. When you're paused
for debugging, there's a console available on the command line that
allows you to re-run tests while keeping breakpoints intact.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#409 (comment)

[nex3]

In the "Debugging" section of the README:

When you hit a breakpoint, the runner will open its own debugging console in the terminal that controls how tests are run. You can type "restart" there to re-run your test as many times as you need to figure out what's going on.

[zoechi]

I'm surprised to hear you say that --pause-after-load feels like a hack. I'm not sure how else you'd support debugging in the test runner.

If you're using Atom without an integrated debugger then this might be fine.
AFAIR debugging in an IDE like WebStorm is not supported because pub run xxx runs xxx in a new process instead of an isolate which prevents attaching a debugger. Actually I'm not sure how this relates to tests run in the browser.

Anyway, I got the impression this is still an open task that needs to be fixed. I don't know where the missing parts are.

IDE integration is something to bring up with IDE authors.

So are you saying that IDE integrated debugging of console and browser tests is not blocked by issues with pub, test, or the Dart VM anymore?

[nex3]

So are you saying that IDE integrated debugging of console and browser tests is not blocked by issues with pub, test, or the Dart VM anymore?

This is true for browser tests, but all debugging of console tests is blocked. See #50.

[TimIrwin26]

So is there any hope for this getting resolved? It's a bit hard for me
to recommend Dart to folks when unit testing is difficult.

I agree with Gunter that while the --pause-after-load option works (in
1.14.2 at least), it is far less than optimal and is an area that has
been a major pain point since the introduction of "pub run test".

On 3/24/16 11:42 AM, Natalie Weizenbaum wrote:

So are you saying that IDE integrated debugging of console and
browser tests is not blocked by issues with |pub|, |test|, or the
Dart VM anymore?

This is true for browser tests, but /all/ debugging of console tests
is blocked. See #50 #50.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#409 (comment)

[nex3]

I hope so, but I can't speak for the IDE integration folks. I'm sure they'd appreciate knowing that this is important to you, although they'd also probably appreciate it in a friendlier tone than you've used here.

[zoechi]

@nex3 thanks a lot for your explanation 👍

[TimIrwin26]

I apologize if my frustration came through. I have 10-15 developers that I'm trying to get started with TDD in our Dart code.

After playing with @zoechi 's suggestion a bit, I found that both script tags can coexist in the .html file, allowing for IDE debugging as a "standard" web page and have "pub run test" work --

<!DOCTYPE html>

<html>
  <head>
    <link rel="x-dart-test" href="file_to_be_tested_test.dart" />
    <script src="packages/test/dart.js"></script>
    <script type="application/dart" src="file_to_be_tested_test.dart"></script>
  </head>
  <body>
    <!-- any html that may be needed for testing can be inserted here -->
  </body>
</html>

[nex3]

I'm pleased that you're unblocked, but be aware that you're using test in a non-standard way. I can't offer any guarantees that that won't be broken in the future due to an implementation change.

[zoechi]

@TimIrwin26 yeah that workaround is actually a hack ;-) (until WebStorm provides better support).
Sorry Natalie for the "hack" for the --pause-after-load. I'm sure it's a solid piece of work and you put lots of effort into it to make it work perfectly - as always 👍