Test generation suite?

[divagant-martian]

I really don't know how would this be done in rust, but I have seen that other tracks generate their tests from the canonical cases. What do you guys think about that?

[petertseng]

It would make following x-common changes easier.

The people making the generators would want to think about two major questions.

• How to read the JSON files, whether that be serde, json-rust, or other.
• Once the files are read, how they turn into tests/*.rs. Templating engine, simple string interpolation, convert to using table-based tests, other.

Additional considerations when writing the generators: The more work each generator is to write, the less likely someone will want to write it.
To reduce the amount of work per generator, common pieces should be factored out. Of course, it may be hard to figure out what the common pieces are before there are at least two generators. That's why this one might not be clear until writing generators. Allow for it in the design if possible.

[ijanos]

I don't think it is feasible to generate full tests because of the lack of types in the JSON files.

A tool that creates some kind of skeleton test file could work or a tool that checks and compares the number of tests in xrust and xcommon. That would make easier to follow changes.

[petertseng]

A tool that creates some kind of skeleton test file could work or a tool that checks and compares the number of tests in xrust and xcommon. That would make easier to follow changes.

If we would include the x-common test version in a standard comment somewhere in xrust, we could compare the xrust vs x-common test versions too.

lack of types in the JSON files.

What of the other languages that have test generators such as Ruby, Go, OCaml, Scala? Surely they dealt with the problem of not having types in the JSON files.

[ijanos]

What of the other languages that have test generators such as Ruby, Go, OCaml, Scala? Surely they dealt with the problem of not having types in the JSON files.

I did a quick check and I found that Scala and OCaml generators both provide a hand-made templates for each exercise. They basically create the types by hand and then the generator builds all the test cases for exercises with templates. This could work with Rust as well, but it is a lot of work and their generators seems language specific (I did not investigate this too much) so I don't think we can use theirs.

If we would include the x-common test version in a standard comment somewhere in xrust, we could compare the xrust vs x-common test versions too.

This is much less work than the other approach and it would help keeping up with the changes in x-common. Implementing this should not be hard, maybe we could create an issue and mark it "good first patch".

[petertseng]

If we would include the x-common test version in a standard comment somewhere in xrust, we could compare the xrust vs x-common test versions too.

This is much less work than the other approach and it would help keeping up with the changes in x-common. Implementing this should not be hard, maybe we could create an issue

Yes, and I have a few questions before it should happen, so I want to talk about them in #281 . Note: it is likely I will do it myself, sorry anyone looking for an easy first patch (there are still a few sitting around so I'm not completely depriving anyone)

[divagant-martian]

Sometimes people just want to help, not to be "easy-first-patch hunters". That said, jsons do not declare types but they use it, integers aren't quoted, strings are, and arrays are as usual delimited by [ ], isn't this enought for most exercises? (at least from the set i've done, i don't remember needing more than that

[ijanos]

Please don't take "easy first patch" as something diminishing. We welcome every contributions and the easy first patch label is there to encourage people.

About the types. Rust has a rich type system, and we try to promote idiomatic rust code. So even the answer is an integer if it represents time for example then the correct type is not just the integer. Similarly strings in json could translate to enum or String or &str in rust. I don't see a way to generalize that without additional metadata.