@Ignore all but the first test case?

[jtigger]

When getting going with an exercise, it can be overwhelming to have a full suite failing all at once. JUnit provides the ability to skip tests through the @Ignore annotation. If all of the tests — save the first one — were skipped, it becomes far easier to see where to get started.

By way of example, here's hello-world, as is (removing the Gradle preamble)...

$ gradle test
...
:test

HelloWorldTest > helloSampleName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:25

HelloWorldTest > helloNullName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:20

HelloWorldTest > helloBlankName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:13

HelloWorldTest > helloNoName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:8

HelloWorldTest > helloAnotherSampleName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:30

5 tests completed, 5 failed
:test FAILED

And here's what it looks like with all but the first test skipped:

$ gradle test
...
:test

HelloWorldTest > helloNoName FAILED
    org.junit.ComparisonFailure at HelloWorldTest.kt:9

5 tests completed, 1 failed, 4 skipped
:test FAILED

[sdavids13]

I don't think it would be obvious for some users to then need to modify the tests to remove the ignores.

[jtigger]

That's a good point.

We're using this technique on the Java track and we haven't had that problem. That said, we also include a play-by-play intro to hello-world that makes plain removing the @Ignore is part of the process.

Are there other ways we could hint for the practitioner to remove the @Ignores one-at-a-time?

[sdavids13]

We also use our tests to verify the example implementations pass too, so that would potentially be problematic. I think what you are kinda going for it to try to mimic ping-pong programming, but do you think it really is overwhelming if they see all fail at once? I really like the documentation that was provided in the Java track's tutorial perhaps that could encourage the practitioner to try to tackle one failure at a time?

[jtigger]

I think what you are kinda going for it to try to mimic ping-pong programming...

That's definitely a part of it, Steve, absolutely: supporting incremental implementation not unlike how ping-pong feels.

... do you think it really is overwhelming if they see all fail at once?

Empathy for less-experienced practitioners is at play. Learning to read error output is enough of a challenge when it is just the one error message, let alone a wall of text and 5-20+ errors. For the those who want to jump in the deep end, mass-removing @Ignore is a minor inconvenience (Ctrl+G in IntelliJ, s/.../g in vim,...) and a small price to pay for accommodating our colleagues.

I really like the documentation that was provided in the Java track's tutorial perhaps that could encourage the practitioner to try to tackle one failure at a time?

The heart of a great UX is delighting our user. In this context, it's about keeping their focus on learning their craft and avoiding burning gumption unnecessarily. UX is so important in @kytrinyx's eyes, she's fundraising for it. And I wouldn't worry that our friends are being coddled, they'll have plenty of other experiences to get frustrated by. :)

If we have the ability, why wouldn't we make the experience more pleasant?

We also use our tests to verify the example implementations pass too, so that would potentially be problematic.

A little Gradle config does the trick:

  task copyTestsFilteringIgnores(type: Copy) {
    from "src/test/xkotlin"
    into "build/gen/test/xkotlin"
    filter { line ->
      line.contains("@Ignore") ? "" : line
    }
  }

  test.dependsOn(copyTestsFilteringIgnores)

alongside a sourceSet definition:

test {
  kotlin.srcDirs = ["build/gen/test/kotlin"]
}

Here's what the build.gradle looks like in the Java track.

And this works swimmingly with IntelliJ's Gradle integration.

[jtigger]

Poking around for a different reason I noticed that in the C# track, they use this same scheme and include a helpful note in the annotation:

https://github.com/exercism/xcsharp/blob/master/exercises/grade-school/GradeSchoolTest.cs

    [Test]
    public void New_school_has_an_empty_roster()
    {
        ...
    }

    [Ignore("Remove to run test")]
    [Test]
    public void Adding_a_student_adds_them_to_the_roster_for_the_given_grade()
    {
        ...
    }

[endofhome]

As a fan of Exercism, someone fairly new to programming and very much hooked on the Kotlin track, I'd like to chime in and say that I always @ignore all but the first test when I begin each exercise. It's not a massive annoyance, but I'd favour that being the default behaviour so I don't have to manually do it.

[jtigger]

Thanks for the from-the-trenches feedback, @forty9er.

Without objection, @sdavids13, I'll start that PR?

[kytrinyx]

We also use our tests to verify the example implementations pass too, so that would potentially be problematic.

@sdavids13 In a lot of the other tracks we have a test script that copies the exercise directory to a temporary directory, rewrites the exercise file to remove the ignore/pending/skip directives, and then run the tests. It's a bit ornery, but was deemed worth it to help smooth the experience for people working on solutions.

[sdavids13]

@jtigger Sounds good! Could you include the tutorial write-up too?

[jtigger]

@sdavids13 asked:

... Sounds good! Could you include the tutorial write-up too?

I'd love to. It's a whole different (and also significant) effort. I'll submit that as a separate PR.

[jtigger]

Addressed by #30