Skip to content

[WIP] Document remote testing #729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 29, 2020
Merged

[WIP] Document remote testing #729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
08fe34d
Document remote testing
tblah May 29, 2020
0b76c52
Document more //ignore-X test headers
tblah May 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/tests/adding.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,8 @@ Some examples of `X` in `ignore-X`:
`musl`.
* Pointer width: `32bit`, `64bit`.
* Stage: `stage0`, `stage1`, `stage2`.
* When cross compiling: `cross-compile`
* When remote testing is used: `remote`

### Other Header Commands

Expand Down
84 changes: 74 additions & 10 deletions src/tests/intro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,81 @@ the Docker image, run it, build Rust within the image, and run the tests.
[src/ci/docker]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
[src/ci/docker/run.sh]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh

## Running tests on a remote machine

Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine
to send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.

To do this, first build `remote-test-server` for the remote
machine, e.g. for RISC-V
```sh
./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```

The binary will be created at
`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
this over to the remote machine.

On the remote machine, run the `remote-test-server` with the `remote` argument
(and optionally `-v` for verbose output). Output should look like this:
```sh
$ ./remote-test-server -v remote
starting test server
listening on 0.0.0.0:12345!
```

You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:
```sh
$ nc $REMOTE_IP 12345
ping
pong
```

To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use
```sh
export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu
```

If `remote-test-server` was run with the verbose flag, output on the test machine
may look something like
```
[...]
run "/tmp/work/test1007/a"
run "/tmp/work/test1008/a"
run "/tmp/work/test1009/a"
run "/tmp/work/test1010/a"
run "/tmp/work/test1011/a"
run "/tmp/work/test1012/a"
run "/tmp/work/test1013/a"
run "/tmp/work/test1014/a"
run "/tmp/work/test1015/a"
run "/tmp/work/test1016/a"
run "/tmp/work/test1017/a"
run "/tmp/work/test1018/a"
[...]
```

Tests are built on the machine running `x.py` not on the remote machine. Tests
which fail to build unexpectedly (or `ui` tests producing incorrect build
output) may fail without ever running on the remote machine.

## Testing on emulators

Some platforms are tested via an emulator for architectures that aren't
readily available. There is a set of tools for orchestrating running the
readily available. For architectures where the standard library is well
supported and the host operating system supports TCP/IP networking, see the
above instructions for testing on a remote machine (in this case the
remote machine is emulated).

There is also a set of tools for orchestrating running the
tests within the emulator. Platforms such as `arm-android` and
`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
emulation on Travis. The following will take a look at how a target's tests
Expand All @@ -165,15 +236,8 @@ reading the results. The Docker image is set up to launch
communicate with the server to coordinate running tests (see
[src/bootstrap/test.rs]).

> TODO: What are the steps for manually running tests within an emulator?
> `./src/ci/docker/run.sh armhf-gnu` will do everything, but takes hours to
> run and doesn't offer much help with interacting within the emulator.
>
> Is there any support for emulating other (non-Android) platforms, such as
> running on an iOS emulator?
>
> Is there anything else interesting that can be said here about running tests
> remotely on real hardware?
> TODO:
> Is there any support for using an iOS emulator?
>
> It's also unclear to me how the wasm or asm.js tests are run.

Expand Down