Broken test-generation for exercise word-count

[georgyo]

This exercise has been marked as having a broken test generator. There are a lots of reasons for test generation to break for a given exercise including

• New addition of new tests that example.ml does not handle correctly. All that is required here is to fix or replace the example!
• Changes to the problem-specification's canonical-data.json that our test-gen cannot handle. This could be then additional of field names, changing field names, or addition of tests that not fit with Ocaml's way of doing things. The task here is figure out what changed in the problem-specification and fix or add rules to special_cases.ml.

Useful links

• Link to broken build output
• Link to example.ml
• Link to canonical-data.json
• Link to problem-specification diff
• Link to special_cases.ml

GitHub doesn't allow for linking the diff of an individual file making the relevant canonical-data.json diff impossible to link to. It may be easier to run this in your terminal

git submodule update --init
git -C problem-specifications diff 2af3c9b0074f16c62366c5c533eaacd3ff27b583 -- exercises/word-count/canonical-data.json

Check List

• Delete templates/word-count/.broken
• Fix test-generation
• Test the test generation with make word-count.gentest

[blastoncrush]

Hello ! I'm really interested into contributing on this project, but I don't know if this issue is still relevant. Is it a good entry point into the dev community ? Thanks by advance !

[kahgoh]

Hi @blastoncrush, yes, this is still an issue. If you'd like to work on this one you'll need to also see how the tests are generated in the README.md. Let me know if you'd like to work on this one or have any further questions.

[blastoncrush]

Hi @kahgoh, I have run all tests on the last build with make test and they all passed, maybe there is something i miss?

[kahgoh]

Did you regenerate the tests after deleting templates/word-count/.broken?

• Delete templates/word-count/.broken
• Regenerate the tests and run the tests as described in the README.md

The tests should fail for word-count. One of the failures should relate to the "with apostrophes" test case - the generator ends up with two cases, but there should only be one since the test case was updated.

[blastoncrush]

Thanks for the response, @kahgoh, now it fails (that sounds weird ik)
So if I understand, the test struggle checking word count with apostrophes somehow?

[blastoncrush]

I got it. On the file exercises/practice/word-count/test.ml, on line 42, the string is written "'First: don't laugh. Then: don't cry. You're getting it.'" with additionnal ' at the beginning and the end. Simply delete then (they were not intended fix the issue

[blastoncrush]

So I fixed problem-specifications/exercises/word-count/canonical-data.json on line 135

[kahgoh]

So I fixed problem-specifications/exercises/word-count/canonical-data.json on line 135

Sorry I think might not have provided enough guidance earlier - there is nothing wrong with the canonical-data.json. Instead, the test generator needs to be updated to handle updated tests.

The canonical-data.json is from the problem specification repository. For practice exercises, this is used to help keep the "common" bits of the exercise in sync across the different tracks. For example, the Java track also base its tests from this same data so that it has similar test cases as OCaml.

However, tests sometimes gets updated. For example, in exercism/problem-specifications#1982, the "with apostrophes" test case was "updated". Notice the one of the "with apostrophes" test case has a reimplements key that references the other test case by its UUID - this indicates that the case with UUID 4ff6c7d7-fcfc-43ef-b8e7-34ff1837a2d3 test case replaces the one with 4185a902-bdb0-4074-864c-f416e42a0f19.

In the OCaml track, we use a test generator to turn the data into tests. I think the real problem to fix here is that the test generator doesn't honor recognize the reimplements key. I haven't fully worked out which files will need to change yet, but this would involve examining the test generator code (in test-generator).

From the sounds of your comment in #471, I suspect updating the test generator to handle the reimplements key might also fix other broken test-generation.