Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Add global test docs. #42531

Merged
merged 2 commits into from
Jun 7, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 124 additions & 3 deletions ci/builders/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,9 +374,130 @@ be relative to the checkout directory.

### Global Tests

Global tests follow the same format as build tests but defined at the top
level. The main difference from local tests is that global tests have access
to the sub-build artifacts.
Tests in this section run on a separate bot as independent sub-builds.
As opposed to tests running within builds, global tests have access to the
the outputs of all the builds running in the same orchestrator build. A
use case for global tests is to run flutter/framework tests using the
artifacts generated by an specific engine build.

Global tests currently support two different scenarios:

* flutter/flutter tests with [tester](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine_v2/tester.py)
recipe. This workflow checks out flutter/flutter to run any of the existing
sharded tests using the engine artifacts archived to GCS.
* complicated engine tests that require the outputs from multiple subbuilds
with [tester_engine](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine_v2/tester_engine.py).
This workflow checks out [flutter/engine] and operates over the dependencies passed to it using cas.

Note: the supported scenarios can be later extended to support running devicelab tests although a
[smart scheduler](https://github.com/flutter/flutter/issues/128294) is a prerequisite for
it to be scalable(build/test separation model).

Framework test example:

```json
{
"tests": [
{
"name": "web-tests-1",
"shard": "web_tests",
"subshard": "1",
"test_dependencies": [
{
"dependency": "chrome_and_driver",
"version": "version:111.0a"
}
]
}
]
}
```

The property's description is as follows:

* **name** the name that will be assigned to the sub-build.
* **shard** the flutter/flutter shard test to run. The supported shard names can be found
on the flutter framework [test.dart](https://github.com/flutter/flutter/blob/master/dev/bots/test.dart#L244).
* **subshard** one of the accepted subshard values for shard. Sub-shards are defined as part
of the shard implementation, please look at the corresponding shard implementation to find the
accepted values.
* **test_dependencies** a list of [dependencies](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipe_modules/flutter_deps/api.py#75)
required for the test to run.

Engine test example:

```json
{
"tests": [
{
"name": "test: lint android_debug_arm64",
"recipe": "engine_v2/tester_engine",
"drone_dimensions": [
"device_type=none",
"os=Linux"
],
"dependencies": [
"host_debug",
"android_debug_arm64"
],
"tasks": [
{
"name": "test: lint android_debug_arm64",
"parameters": [
"--variant",
"android_debug_arm64",
"--lint-all",
"--shard-id=0",
"--shard-variants=host_debug"
],
"max_attempts": 1,
"script": "flutter/ci/lint.sh"
}
]
}
]
}
```

The property's description is as follows:

* **name** the name to assign to the sub-build.
* **recipe** the recipe name to use if different than tester.
* **drone_dimensions** a list of strings with key values to select the
bot where the test will run.
* **dependencies** a list of build outputs required
by the test. These build outputs are referenced by the name of build
generating the output. This type of dependency is shared using CAS and
the contents are mounted in checkout/src/out. E.g. a build configuration
building the `host_engine` configuration will upload the content of
checkout/src/out/host_engine to CAS and a global test with a `host_engine`
dependency will mount the content of host engine in the same location of
the bot running the test.
* **tasks** a list of dictionaries representing scripts and parameters to run them.

Example task configuration:

```json
{
"name": "test: lint android_debug_arm64",
"parameters": [
"--variant",
"android_debug_arm64",
"--lint-all",
"--shard-id=0",
"--shard-variants=host_debug"
],
"max_attempts": 1,
"script": "flutter/ci/lint.sh"
}
```

The property's description is as follows:

* **name** the name assigned to the step running the script.
* **parameters** a list of parameters passed to the script execution.
* **max_attempts** an integer with the maximum number of runs in case of failure.
* **script** the path relative to checkout/src/ to run.

### Global Generators

Expand Down