Test suite

[ehuss]

I think it would be very helpful to have a test suite integrated with the spec.

Benefits

There are several reasons:

• It serves as a validation for changes or regressions in rustc that may get out of sync with the spec.
• It helps illustrate what the rules are trying to convey. This can service several audiences:
  • The spec author. During the process of writing the text, I have found that I usually write a large series of tests to validate my understanding.
  • The spec reviewer. This can help the reviewer validate their own understanding matches the code to the text, and to help check how complete the coverage is.
  • The spec reader. People often skip to examples first rather than reading the text, since they can understand things more quickly that way. Additionally, tests can help them validate their understanding of tricky concepts.

As an example, Ferrocene has a Traceability matrix report that shows connects to the rustc testsuite to sections within the spec. The Reference uses the rustdoc test functionality to verify examples within the source text.

Where are the tests?

My suggestion would be to have an independent test suite that lives with the spec (instead of, for example, linking to rustc's test suite). That has a few benefits:

• The tests can be specifically tailored to illustrate concepts within the spec. Some rustc tests may not directly demonstrate some narrow concept or rule.
• The tests can include comments and style that is approachable to a typical Rust programmer. Some rustc tests can be very terse, or unclear what they are testing.
• More easily add tests. Pushing to a separate repository can be a pain.
• Easier to review changes and additions to the spec. With the tests within a PR, the reviewer can directly see what is being changed without referencing an external site.
• Prevents tests from getting out of sync. If rustc changes, renames, or removes tests, the spec will be constantly needing to chase those down. Additionally, a test could be changed to not cover what the spec expects it to cover.
• Provides an independent validation of regressions in rustc. This has happened a very small number of times in the past.

Of course the major downside is actually writing the tests. However, I expect that authors will need to write test code anyways to verify their own understanding.

Viewing the tests

I would recommend making it easy to view the test files from the spec. I don't know exactly what that would look like, but maybe there could be some kind of link or icon next to each rule name, and that takes you to the tests linked with that rule? This will be hard to not make it too noisy.

Inline or outline tests

I expect there will be two kinds of tests:

• Simple illustrations shown within the text to help the reader see some examples.
• More complex or exhaustive tests to validate the behavior of specific rules, but not necessarily shown directly in the text.

"Inline" and "outline" have different meanings in different contexts:

• How you write the code: Code samples to be shown to the user in the text can be inline via markdown code blocks, or outline via mdbook includes.
• How the reader sees the code: Code that validates a rule can be shown inline in the text to the user (as illustrations), or can be placed in a separate test suite which requires the user to take some action to view (like the FLS test matrix).

Inline test benefits:

• Easier to author. No need to create another file, think of a name for it, etc.
• Easier to review. GitHub's PR preview would not show outline tests, and it can be difficult to correlate which test is being shown.

Outline test benefits:

• Support multiple files and other more complex build requirements.
• Easier to show a subset of a large code sample (using include "anchors").

I would recommend integrating the test suite with inline examples, to make it easier to author and review simple code snippets. Larger examples should use mdbook includes if desired (see how TRPL does this, and #19). However, most rules should not have their tests displayed in the text, but instead accessed through some relatively easy means (like a link). #19 covers what the guidelines should be for showing in-text examples.

Tooling

I would expect the tests to be roughly the same as what rustc uses. Whether that is ui_test or compiletest, I don't know. I'm only vaguely aware of ui_test, so I'm not sure what its limitations are.

[mattheww]

For the lexer, recently I've been testing by linking to rustc using rustc_private and looking at the tokens being fed to the parser.

I don't think that would be a sensible thing for a testsuite in the spec to do, but I've found it a lot more practical than testing via tt by-example macros (especially for testing cases that are rejected). In particular it shows how each token was classified, not just which characters it contained.

Maybe it would be thinkable for rustc to add a -Z option that reports the output of the lexer (roughly what rustc_parse::parser::Parser bump() gives), and for the spec to use that in a testsuite?

I suppose there's a sense in which anything more internal than "does this program compile?" is out of scope for the spec, but pragmatically if we're going to try to document the lexer as a stage of its own it would be nice to have a way to test what the spec ends up saying directly.

[JoelMarcey]

After the t-spec meeting today, the general consensus on how we approach the test suite is that we utilize existing rustc tests (adding where necessary) for coverage and validation of spec content. This will take the shape of either adding tags in the existing tests for the spec and/or creating a separate subtree in the test suite for the specification tests. For illustrative tests, for example inline examples within the specification, the specification will manage those in house.

Conversation here: https://hackmd.io/jR6-og7YQ9WFIz1wpaDqhw

[ehuss]

Update on this:

• We shifted the plan to keep most of the tests in rust-lang/rust relying on compiletest and the infrastructure there. Test rule annotations compiler-team#783 details the plan where rules are associated with tests via annotations.
• Tests are now linked from the reference (though most rules are not updated). See https://doc.rust-lang.org/nightly/reference/input-format.html as an example.
• The reference has a summary table published here: https://doc.rust-lang.org/nightly/reference/test-summary.html

I still think there is some work to do here.

• There's a clumsy process for associating tests for unstable features.
• It's not really clear that this can be integrated into the review process in a useful way.
• The difference between inline and outline tests isn't clear.
• The inline tests relying on rustdoc are limited.
• No process for fixing broken test associations.
• It can sometimes be difficult to find the correct tests in rust-lang/rust, or there are no tests at all.
• Not clear how to handle non-terminal rules. For example with rule foo.bar.baz, should it be allowed to annotate foo.bar? What does that even mean?