Schema optimises for testing `f(inputs) = output`. What of other types of tests?

[petertseng]

Our JSON schema right now most ergonomically expresses tests of the form:

The code under test, given this input, produces this exact output.

This is expressed in https://github.com/exercism/problem-specifications/blob/master/canonical-schema.json#L6-L8, which today reads:

   , " optimising for the ability to represent example-based tests"
   , " of the form 'function (input) == output'. Future expansions"
   , " may allow for other types of tests such as property-based. "

What other types might we care about:

• The code under test, given X input, produces the same result as when it is given Y input. In other words, f(X) == f(Y)
  • diffie-hellman: Add canonical data #1224 (comment): diffie-hellman key exchange should result in two parties calculating the same secret
  • zipper: add canonical data #934 (comment) and zipper: add canonical data #934 (comment): You can arrive at equal zippers using two different series of operations.
• Create a thing then perform some number of operations on it, each of which should produce some expected result
  • Circular-buffer circular-buffer: Add canonical data #488
  • React react: Add JSON test data #358
  • possibly bank-account, pending JSON: bank-account: Implement canonical-data.json #554
• Property-based tests
  • diamond for sure
  • possibly "answer is within this range" such as diffie-hellman's diffie-hellman: Add canonical data #1224 (comment)
  • possibly "answer matches this regular expression" such as simple-cipher's simple-cipher: Add canonical data #966 (comment)
  • possibly "answer is random" such as diffie-hellman's diffie-hellman: Add canonical data #1224 (comment), simple-cipher's simple-cipher: Add canonical data #966 (comment), and robot-name
• The input to a function is the output of another function, perhaps f(g(x))
  • simple-cipher with its tests that decode . encode and encode . decode are both identity, as described in simple-cipher: Add canonical-data.json #1241 (comment)
• Certain strings (in either the input or output) are to be interpreted not as their literal value but in some symbolic way
  • simple-cipher with things like cipher.key as in simple-cipher: Add canonical-data.json #1241 (comment)

The number of uses for each of these alternate test types is not large enough that I feel it is worth any possible proposal at this time. Therefore I'm closing this issue immediately and taking no action on it.

If you want to make it easy to make a proposal when the time is right for it, it is in your interest to continue reporting any such use cases here.

[coriolinus]

It feels to me like expanding into any of those categories puts us squarely into property-based testing territory.

All property based test frameworks with which I am familiar (1, 2, 3) have a programmatic interface. Expressing a property based test suitable for multiple languages via a JSON file sounds would be an ambitious undertaking. I suspect that for our cases, we'll do better to construct function-based tests for the canonical data, and write comments in English about particular property-based tests which tracks may wish to implement manually.

[cmccandless]

@coriolinus what about adding a "implementManually" = true or "noAuto" = true tag for cases that may require human implementation?

[coriolinus]

I prefer comments.

Some test generators written before inputs were standardized infer the problem inputs: everything which isn't a known field must be an input. "comments" and "input" are known fields, so nothing is broken as yet.

Adding a new field breaks those generators.

[SleeplessByte]

Adding a new field breaks those generators.

Yeah I don't think this is a good reason to not add new fields. I actually think it's one of the worst reasons. In this case, we can reduce the workload for all tracks significantly by adding new properties, and perhaps making the schema versioned.

This is a one time cost for generators, which just need to ignore properties, and adds a lot of benefit instantly and in the future.

[sshine]

I also prefer comments to describe properties for the reason given by @coriolinus:

Because property based test frameworks [...] have a programmatic interface.

Edit: Clarified reason, as @coriolinus gave two.