Skip to content

Commit beb2d89

Browse files
nikomatsakisnikomatsakis
committed
document the test infrastructure
1 parent 6ebe10d commit beb2d89

File tree

5 files changed

+389
-2
lines changed

5 files changed

+389
-2
lines changed

‎src/SUMMARY.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,9 @@
22

33
- [About this guide](./about-this-guide.md)
44
- [How to build the compiler and run what you built](./how-to-build-and-run.md)
5-
- [Using the compiler testing framework](./running-tests.md)
5+
- [The compiler testing framework](./tests/intro.md)
6+
- [Running tests](./tests/running.md)
7+
- [Adding new tests](./tests/adding.md)
68
- [Walkthrough: a typical contribution](./walkthrough.md)
79
- [High-level overview of the compiler source](./high-level-overview.md)
810
- [Queries: demand-driven compilation](./query.md)

‎src/running-tests.md

-1
This file was deleted.

‎src/tests/adding.md

+273
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
# Adding new tests
2+
3+
**In general, we expect every PR that fixes a bug in rustc to come
4+
accompanied with a regression test of some kind.** This test should
5+
fail in master but pass after the PR. These tests are really useful
6+
for preventing us from repeating the mistakes of the past.
7+
8+
To add a new test, the first thing you generally do is to create a
9+
file, typically a Rust source file. Test files have a particular
10+
structure:
11+
12+
- They always begin with the copyright notice;
13+
- then they should have some kind of
14+
[comment explaining what the test is about](#explanatory_comment);
15+
- next, they can have one or more [header commands](#header_commands), which are special
16+
comments that the test interpreter knows how to interpret.
17+
- finally, they have the Rust source. This may have various [error
18+
annotations](#error_annotations) which indicate expected compilation errors or
19+
warnings.
20+
21+
Depending on the test suite, there may be some other details to be aware of:
22+
- For [the `ui` test suite](#ui), you need to generate reference output files.
23+
24+
## Naming your test
25+
26+
We have not traditionally had a lot of structure in the names of
27+
tests. Moreover, for a long time, the rustc test runner did not
28+
support subdirectories (it now does), so test suites like
29+
`src/test/run-pass` have a huge mess of files in them. This is not
30+
considered an ideal setup.
31+
32+
For regression tests -- basically, some random snippet of code that
33+
came in from the internet -- we often just name the test after the
34+
issue. For example, `src/test/run-pass/issue-1234.rs`. If possible,
35+
though, it is better if you can put the test into a directory that
36+
helps identify what piece of code is being tested here (e.g.,
37+
`borrowck/issue-12345.rs` is much better), or perhaps give it a more
38+
meaningful name. Still, **do include the issue number somewhere**.
39+
40+
When writing a new feature, **create a subdirectory to store your
41+
tests**. For example, if you are implementing RFC 1234 ("Widgets"),
42+
then it might make sense to put the tests in directories like:
43+
44+
- `src/test/ui/rfc1234-widgets/`
45+
- `src/test/run-pass/rfc1234-widgets/`
46+
- etc
47+
48+
In other cases, there may already be a suitable directory. (The proper
49+
directory structure to use is actually an area of active debate.)
50+
51+
<a name=explanatory_comment>
52+
53+
## Comment explaining what the test is about
54+
55+
When you create a test file, **include a comment summarizing the point
56+
of the test immediately after the copyright notice**. This should
57+
highlight which parts of the test are more important, and what the bug
58+
was that the test is fixing. Citing an issue number is often very
59+
helpful.
60+
61+
This comment doesn't have to be super extensive. Just something like
62+
"Regression test for #18060: match arms were matching in the wrong
63+
order." might already be enough.
64+
65+
These comments are very useful to others later on when your test
66+
breaks, since they often can highlight what the problem is. They are
67+
also useful if for some reason the tests need to be refactored, since
68+
they let others know which parts of the test were important (often a
69+
test must be rewritten because it no longer tests what is was meant to
70+
test, and then it's useful to know what it *was* meant to test
71+
exactly).
72+
73+
<a name=header_commands>
74+
75+
## Header commands: configuring rustc
76+
77+
Header commands are special comments that the test runner knows how to
78+
interpret. They must appear before the Rust source in the test. They
79+
are normally put after the short comment that explains the point of
80+
this test. For example, this test uses the `// compile-flags` command
81+
to specify a custom flag to give to rustc when the test is compiled:
82+
83+
```rust
84+
// Copyright 2017 The Rust Project Developers. blah blah blah.
85+
// ...
86+
// except according to those terms.
87+
88+
// Test the behavior of `0 - 1` when overflow checks are disabled.
89+
90+
// compile-flags: -Coverflow-checks=off
91+
92+
fn main() {
93+
let x = 0 - 1;
94+
...
95+
}
96+
```
97+
98+
### Ignoring tests
99+
100+
These are used to ignore the test in some situations, which means the test won't
101+
be compiled or run.
102+
103+
* `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below)
104+
* `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work)
105+
* `ignore-test` always ignores the test
106+
* `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger.
107+
108+
Some examples of `X` in `ignore-X`:
109+
110+
* Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ...
111+
* OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ...
112+
* Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`.
113+
* Pointer width: `32bit`, `64bit`.
114+
* Stage: `stage0`, `stage1`, `stage2`.
115+
116+
### Other Header Commands
117+
118+
Here is a list of other header commands. This list is not
119+
exhaustive. Header commands can generally be found by browsing the
120+
`TestProps` structure found in [`header.rs`] from the compiletest
121+
source.
122+
123+
* `min-{gdb,lldb}-version`
124+
* `min-llvm-version`
125+
* `must-compile-successfully` for UI tests, indicates that the test is supposed
126+
to compile, as opposed to the default where the test is supposed to error out.
127+
* `compile-flags` passes extra command-line args to the compiler,
128+
e.g. `compile-flags -g` which forces debuginfo to be enabled.
129+
* `should-fail` indicates that the test should fail; used for "meta testing",
130+
where we test the compiletest program itself to check that it will generate
131+
errors in appropriate scenarios. This header is ignored for pretty-printer tests.
132+
* `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X.
133+
Such tests are supposed to ensure that the compiler errors when usage of a gated
134+
feature is attempted without the proper `#![feature(X)]` tag.
135+
Each unstable lang feature is required to have a gate test.
136+
137+
[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs
138+
139+
<a name="error_annotations">
140+
141+
## Error annotations
142+
143+
Error annotations specify the errors that the compiler is expected to
144+
emit. They are "attached" to the line in source where the error is
145+
located.
146+
147+
* `~`: Associates the following error level and message with the
148+
current line
149+
* `~|`: Associates the following error level and message with the same
150+
line as the previous comment
151+
* `~^`: Associates the following error level and message with the
152+
previous line. Each caret (`^`) that you add adds a line to this, so
153+
`~^^^^^^^` is seven lines up.
154+
155+
The error levels that you can have are:
156+
157+
1. `ERROR`
158+
2. `WARNING`
159+
3. `NOTE`
160+
4. `HELP` and `SUGGESTION`*
161+
162+
\* **Note**: `SUGGESTION` must follow immediately after `HELP`.
163+
164+
## Revisions
165+
166+
Certain classes of tests support "revisions" (as of the time of this
167+
writing, this includes run-pass, compile-fail, run-fail, and
168+
incremental, though incremental tests are somewhat
169+
different). Revisions allow a single test file to be used for multiple
170+
tests. This is done by adding a special header at the top of the file:
171+
172+
```
173+
// revisions: foo bar baz
174+
```
175+
176+
This will result in the test being compiled (and tested) three times,
177+
once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
178+
baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak
179+
each of these results.
180+
181+
You can also customize headers and expected error messages to a particular
182+
revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//`
183+
comment, like so:
184+
185+
```
186+
// A flag to pass in only for cfg `foo`:
187+
//[foo]compile-flags: -Z verbose
188+
189+
#[cfg(foo)]
190+
fn test_foo() {
191+
let x: usize = 32_u32; //[foo]~ ERROR mismatched types
192+
}
193+
```
194+
195+
Note that not all headers have meaning when customized to a revision.
196+
For example, the `ignore-test` header (and all "ignore" headers)
197+
currently only apply to the test as a whole, not to particular
198+
revisions. The only headers that are intended to really work when
199+
customized to a revision are error patterns and compiler flags.
200+
201+
<a name="ui">
202+
203+
## Guide to the UI tests
204+
205+
The UI tests are intended to capture the compiler's complete output,
206+
so that we can test all aspects of the presentation. They work by
207+
compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output,
208+
and then applying some normalization (see below). This normalized
209+
result is then compared against reference files named
210+
`ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of
211+
those files doesn't exist, the output must be empty. If the test run
212+
fails, we will print out the current output, but it is also saved in
213+
`build/<target-triple>/test/ui/hello_world/main.stdout` (this path is
214+
printed as part of the test failure message), so you can run `diff` and
215+
so forth.
216+
217+
Normally, the test-runner checks that UI tests fail compilation. If you want
218+
to do a UI test for code that *compiles* (e.g. to test warnings, or if you
219+
have a collection of tests, only some of which error out), you can use the
220+
`// must-compile-successfully` header command to have the test runner instead
221+
check that the test compiles successfully.
222+
223+
### Editing and updating the reference files
224+
225+
If you have changed the compiler's output intentionally, or you are
226+
making a new test, you can use the script `ui/update-references.sh` to
227+
update the references. When you run the test framework, it will report
228+
various errors: in those errors is a command you can use to run the
229+
`ui/update-references.sh` script, which will then copy over the files
230+
from the build directory and use them as the new reference. You can
231+
also just run `ui/update-all-references.sh`. In both cases, you can run
232+
the script with `--help` to get a help message.
233+
234+
### Normalization
235+
236+
The normalization applied is aimed at eliminating output difference
237+
between platforms, mainly about filenames:
238+
239+
- the test directory is replaced with `$DIR`
240+
- all backslashes (`\`) are converted to forward slashes (`/`) (for Windows)
241+
- all CR LF newlines are converted to LF
242+
243+
Sometimes these built-in normalizations are not enough. In such cases, you
244+
may provide custom normalization rules using the header commands, e.g.
245+
246+
```
247+
// normalize-stdout-test: "foo" -> "bar"
248+
// normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)"
249+
// normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
250+
```
251+
252+
This tells the test, on 32-bit platforms, whenever the compiler writes
253+
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
254+
instead. Similar for 64-bit. The replacement is performed by regexes using
255+
default regex flavor provided by `regex` crate.
256+
257+
The corresponding reference file will use the normalized output to test both
258+
32-bit and 64-bit platforms:
259+
260+
```
261+
...
262+
|
263+
= note: source type: fn() ($PTR bits)
264+
= note: target type: u16 (16 bits)
265+
...
266+
```
267+
268+
Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example.
269+
270+
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
271+
information or stage supported by `ignore-X` here as well (e.g.
272+
`normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional
273+
replacement).

‎src/tests/intro.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Using the compiler testing framework
2+
3+
The compiler has an extensive testing framework, masterminded by the
4+
compiletest tool (sources in the [`src/tools/compiletest`]). This
5+
section gives a brief overview of how the testing framework is setup,
6+
and then gets into some of the details on
7+
[how to run tests](running.html) as well as
8+
[how to add new tests](adding.html).
9+
10+
[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
11+
12+
## Test suites
13+
14+
The tests are located in the tree in the [`src/test`]
15+
directory. Immediately within you will see a series of subdirectories
16+
(e.g. `ui`, `run-make`, and so forth). Each of those directories is
17+
called a **test suite** -- they house a group of tests that are run in
18+
a distinct mode.
19+
20+
[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test
21+
22+
Here is a brief summary of the test suites as of this writing and what
23+
they mean. In some cases, the test suites are linked to parts of the manual
24+
that give more details.
25+
26+
- [`ui`](ui.html) -- tests that check the exact stdout/stderr from compilation
27+
and/or running the test
28+
- `run-pass` -- tests that are expected to compile and execute successfully (no panics)
29+
- `run-pass-valgrind` -- tests that ought to run with valrind
30+
- `run-fail` -- tests that are expected to compile but then panic during execution
31+
- `compile-fail` -- tests that are expected to fail compilation.
32+
- `parse-fail` -- tests that are expected to fail to parse
33+
- `pretty` -- tests targeting the Rust "pretty printer", which
34+
generates valid Rust code from the AST
35+
- `debuginfo` -- tests that run in gdb or lldb and query the debug info
36+
- `codegen` -- tests that compile and then test the generated LLVM
37+
code to make sure that optimizing we want are kicking in etc
38+
- `mir-opt` -- tests that check parts of the generated MIR to make sure we are optimizing
39+
etc.
40+
- `incremental` -- tests for incremental compilation, checking that
41+
when certain modifications are performed, we are able to reuse the
42+
results from previous compilations.
43+
- `run-make` -- tests that basically just execute a `Makefile`; the ultimate in flexibility
44+
but annoying as all get out to write.
45+
- `rustdoc` -- tests for rustdoc, making sure that the generated files contain
46+
documentation for various entities etc
47+
- `*-fulldeps` -- same as above, but indicates that the test depends on things other
48+
than `libstd` (and hence those things must be built)

0 commit comments

Comments
 (0)