Test order is affected by a presence of a nested test

[ArtemGovorov]

The following code:

var test = require('tape');

test('first', function (t) {

  t.plan(2);

  setTimeout(function () {
    t.ok(1, 'first test');
  }, 200);

  t.test('second', function (t) {
    t.ok(1, 'second test');
    t.end();
  });
});

test('third', function (t) {
  setTimeout(function () {
    t.ok(1, 'third test');
    t.end();
  }, 100);
});

outputs:

ok 1 first test
ok 2 second test
ok 3 third test

and it's expected. But if I change it to its logical equivalent without plan:

var test = require('tape');

test('first', function (t) {

  setTimeout(function () {
    t.ok(1, 'first test');
    t.end();
  }, 200);

  t.test('second', function (t) {
    t.ok(1, 'second test');
    t.end();
  });
});

test('third', function (t) {
  setTimeout(function () {
    t.ok(1, 'third test');
    t.end();
  }, 100);
});

the the test order is broken:

ok 1 second test
ok 2 third test
ok 3 first test

While changing the order of the first and the second test is somewhat expected, ending the first test prematurely, and letting the third test execute before it, looks like a bug. Especially considering that commenting t.test('second'... test out restores the expected order, and make the third test wait until the first one ends.

[davegomez]

Is anyone checking at this? @substack @ljharb

[GuidoS]

I think this may be tied to this other open issue #122

[Raynos]

This is not an ordering problem; this is a reporting problem.

t.end() will wait for all child tests to finish.

I agree this is confusing semantic difference between t.end() and t.plan().

I recommend adding console.log() to convince yourself of the actual order of execution and to not worry about which order tape reports TAP statements to stdout.

[ArtemGovorov]

@Raynos From what I see, it is definitely an ordering issue, not a reporting one.
Adding console.log (with timestamps just in case), produces the same result:

var test = require('tape');

test('first', function (t) {

  setTimeout(function () {
    console.log('first test', +new Date());
    t.ok(1);
    t.end();
  }, 200);

  t.test('second', function (t) {
    console.log('second test', +new Date());
    t.ok(1);
    t.end();
  });
});

test('third', function (t) {
  setTimeout(function () {
    console.log('third test', +new Date());
    t.ok(1);
    t.end();
  }, 100);
});

second test 1451094351565
third test 1451094351676
first test 1451094351756

[Raynos]

@ArtemGovorov

Your right; this is simply broken.

[Raynos]

I honestly do not know enough about what the semantics of sub tests are supposed to be to fix this...

It should either be:

second test
first test
third test

or

first test
second test
third test

[ArtemGovorov]

I think it should be

first test
second test
third test

because sub tests are not much different from the same level tests in a sense that they may need the previous/parent test to be finished. For example, nested tests may use a context that is set up in the parent test and may need the parent test to finish working with the context before a nested test can start using the context.

[ljharb]

I would expect second, first, third, based on my expectation of subtest ordering - the setTimeout means that t.end doesn't run on the current tick, so all the subtests of "first test" would run before the next tick starts, and "first test" gets logged.

tape follows JS scoping rules, so the ordering of the tests shouldn't affect anything unless a subtest mutates something another subtest depends on.

[ArtemGovorov]

@ljharb Why allowing the second test to run before the first one is finished? Except the nesting level, the second and the third tests are not much different conceptually, so why having different rules for the sub tests?

[ArtemGovorov]

In other words, the first test may be doing things in this scope:

test('first', function (t) {
  ...

as well as in this scope:

  setTimeout(function () {
     ...
  }, 200);

Allowing the second test to run before the first one is finished, means potentially interrupting the work of the first test. The isolation of the first test is broken.

[ljharb]

Using setTimeout in this fashion seems very strange to me - you're actually delaying creation of the subtest. If you do the setTimeout inside the subtest, I suspect you'll be much less surprised by the test ordering. Can you elaborate on why you're doing that?

[Raynos]

The bug here is that sub tests work differently purely on the existence of plan()

[ArtemGovorov]

@ljharb So I had a code like this:

var test = require('tape');

test('first', function (t) {

  setTimeout(function () {
    t.ok(1, 'first test');
    t.end();
  }, 200);
});

test('third', function (t) {
  setTimeout(function () {
    t.ok(1, 'third test');
    t.end();
  }, 100);
});

The first test was finishing before the third one. Then I added a subtest:

var test = require('tape');

test('first', function (t) {

  setTimeout(function () {
    t.ok(1, 'first test');
    t.end();
  }, 200);

  t.test('second', function (t) {
    t.ok(1, 'second test');
    t.end();
  });
});

test('third', function (t) {
  setTimeout(function () {
    t.ok(1, 'third test');
    t.end();
  }, 100);
});

Adding the second test introduced 2 issues:

• the second test (subtest) is executed before the first test is finished;
• the third test is now executed before the first test is finished.

Regarding the first issue - semantically, subtests are separate tests. They're executed after their parent test. That's what happens when the parent test in synchronous or has .plan call like here. Async parent test without the explicit .plan call behaviour is just inconsistent with the rest.

If you do the setTimeout inside the subtest

I know I may do the setTimeout inside the subtest, or may define the subtest inside the setTimeout, or may add .plan just like in the example I gave in the initial issue report. The issue with inconsistent behaviour remains though, as @Raynos said, and I agree, the bug is that sub tests work differently purely on the existence of plan().

Regarding the second issue - no matter how I structure the first test subtests, the third test should never execute before the first one is finished. It is inconsistent with tape's promise to execute tests sequentially.

[ljharb]

Gotcha - I always use plan so I haven't ran into it. I agree that the existence of plan shouldn't affect ordering of the tests.