Skip to content

Commit 91d482e

Browse files
Merge pull request #1269 from mintlayer/fix/coverage2
Partition coverage into multiple jobs for diskspace in CI
2 parents af63ce2 + a13597c commit 91d482e

File tree

4 files changed

+154
-13
lines changed

4 files changed

+154
-13
lines changed

.github/workflows/coverage.yml

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -8,39 +8,43 @@ on:
88

99
name: Code Coverage
1010

11+
env:
12+
# We partition coverage tests into multiple parts to avoid filling diskspace in a single runner
13+
PARTITIONS_COUNT: 5
14+
1115
jobs:
1216
coverage:
1317
runs-on: ubuntu-latest
18+
19+
strategy:
20+
matrix:
21+
# This range spans from `0` to `PARTITIONS_COUNT - 1`, where `PARTITIONS_COUNT` is the number of partitions (defined in env var above)
22+
partition: [0, 1, 2, 3, 4]
23+
1424
steps:
15-
- id: skip-check
16-
uses: fkirc/[email protected]
17-
with:
18-
concurrent_skipping: "same_content_newer"
19-
skip_after_successful_duplicate: "true"
2025
- name: Install dependencies
2126
run: sudo apt-get install -yqq --no-install-recommends build-essential libgtk-3-dev python3 python3-toml
2227
- uses: actions/checkout@v1
2328
- uses: actions-rs/toolchain@v1
2429
with:
25-
# TODO: Inspect coverage, and change toolchain to stable
2630
toolchain: nightly-2023-08-01
2731
override: true
2832
- uses: actions-rs/cargo@v1
2933
with:
3034
command: clean
31-
- uses: actions-rs/cargo@v1
32-
with:
33-
command: test
34-
args: --all-features --no-fail-fast
35+
- name: Run coverage tests
36+
run: python3 build-tools/workspace-partition.py ${{ env.PARTITIONS_COUNT }} ${{ matrix.partition }} -p | xargs cargo test
3537
env:
3638
RUST_LOG: debug
3739
RUST_BACKTRACE: full
3840
CARGO_INCREMENTAL: 0
39-
RUSTFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests"
40-
RUSTDOCFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests"
41+
RUSTFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off"
42+
RUSTDOCFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off"
4143
- id: coverage
4244
uses: actions-rs/[email protected]
45+
with:
46+
config: build-tools/coverage/grcov.toml
4347
- uses: actions/upload-artifact@v2
4448
with:
45-
name: code-coverage-report
49+
name: code-coverage-report-${{ matrix.partition }}
4650
path: ${{ steps.coverage.outputs.report }}

build-tools/coverage/README.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Generating test coverage report
2+
3+
Using the configuration in this directory, Github Actions CI will generate a test coverage report for the whole project. The report possibly will be split into multiple parts due to disk-space constraints related to github CI limits. This tutorial describes how to generate an html report out of this data.
4+
5+
## Prerequisites
6+
- The source code of the project
7+
- The coverage report data generated by the CI
8+
- lcov installed, together with genhtml.
9+
10+
## Steps
11+
1. Download the reports data from CI. The data is stored in the artifacts of the CI job. You can download it manually from the CI job page.
12+
2. Unzip all the downloaded files into a single directory. They all will have the prefix `grcov-report-`.
13+
3. Now we need to run the following command with all the files in the directory:
14+
15+
```bash
16+
lcov -a grcov-report-1 -a grcov-report-2 -a grcov-report-3 -a grcov-report-4 -a grcov-report-5 -o grcov-report.info
17+
```
18+
19+
where these suffixes can be different every time. The easiest way is to run this magic bash command which will concatenate all the files in the directory, prefix them with `-a`, and pass them to lcov:
20+
21+
```bash
22+
ls -1a grcov-report-* | sed 's/^/-a /' | tr '\n' ' ' | xargs lcov -o grcov-report.info
23+
```
24+
25+
4. Now we need to generate the html report. Go to the source code's root dir (and note where you put that grcov-report.info file)
26+
27+
5. run the following command:
28+
29+
```bash
30+
genhtml --exclude /rustc/* --exclude */.cargo/registry/* --exclude */.cargo/git/* --exclude /cargo/registry/* --exclude "*target/*" -o grcov-report --ignore-errors unmapped /path/to/grcov-report.info
31+
```
32+
33+
6. Now you can open the `grcov-report/index.html` file in your browser and see the report.

build-tools/coverage/grcov.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
output-type: lcov
2+
output-path: ./lcov.info

build-tools/workspace-partition.py

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
import sys
2+
import toml
3+
import os
4+
5+
def read_cargo_toml(file_path):
6+
with open(file_path, "r") as cargo_toml_file:
7+
cargo_toml = toml.load(cargo_toml_file)
8+
return cargo_toml
9+
10+
def get_package_name(crate_dir):
11+
cargo_toml_path = os.path.join(crate_dir, "Cargo.toml")
12+
cargo_toml = read_cargo_toml(cargo_toml_path)
13+
14+
if cargo_toml is not None:
15+
package = cargo_toml.get("package", {})
16+
return package.get("name", None)
17+
else:
18+
raise Exception("Cargo.toml not found for crate {}.".format(crate_dir))
19+
20+
def group_crates_by_first_directory(members):
21+
'''
22+
Here we ensure that all crates in the same directory are grouped together, to minimize fracturing of coverage.
23+
We assume here that crates are tested within their directory. This is a fair assumption since we put test-suites
24+
in the same directory as the crate they are testing.
25+
'''
26+
crate_groups = {}
27+
for crate_dir in members:
28+
# Split the path name with the directory separator
29+
dir_parts = crate_dir.split(os.path.sep)
30+
if dir_parts:
31+
# Group crates by the first element and join them back using the separator
32+
dir_name = os.path.sep.join(dir_parts[:1])
33+
if dir_name not in crate_groups:
34+
crate_groups[dir_name] = []
35+
crate_groups[dir_name].append(crate_dir)
36+
return crate_groups
37+
38+
def partition_workspace(m, n):
39+
cargo_toml = read_cargo_toml("Cargo.toml")
40+
41+
if cargo_toml is None:
42+
raise Exception("Cargo.toml not found.")
43+
44+
workspace = cargo_toml.get("workspace", {})
45+
members = workspace.get("members", [])
46+
47+
if not members:
48+
raise Exception("No members found in the workspace.")
49+
50+
# Group crates based on directory structure using directory separators
51+
crate_directory_groups = group_crates_by_first_directory(members)
52+
53+
# Calculate elements per partition and remainder based on the number of crate directories
54+
total_directories = len(crate_directory_groups)
55+
elements_per_partition = total_directories // m
56+
remainder = total_directories % m
57+
58+
# Calculate the start and end indices for the current partition
59+
start_idx = n * elements_per_partition + min(n, remainder)
60+
end_idx = start_idx + elements_per_partition + (1 if n < remainder else 0)
61+
62+
# Get the crate directories for the current partition
63+
partition_directories = []
64+
for dir_path in list(crate_directory_groups.keys())[start_idx:end_idx]:
65+
partition_directories.extend(crate_directory_groups[dir_path])
66+
67+
# Get the package names from the crate directories in this partition
68+
package_names = [get_package_name(crate_dir) for crate_dir in partition_directories]
69+
70+
return package_names
71+
72+
if __name__ == "__main__":
73+
if len(sys.argv) == 2 and (sys.argv[1] == "--help" or sys.argv[1] == "-h"):
74+
print("Usage: python partition_workspace.py <total_partitions> <partition_index> [prefix]")
75+
print("")
76+
print("Partitions the workspace into m partitions and returns the crates in the n-th partition.")
77+
print("This can be used to split the workload of running tests across multiple CI jobs.")
78+
print("")
79+
print("To run the tests for the n-th partition, use the following command (for 3 partitions):")
80+
print("---")
81+
print("python3 {} 3 0 -p | xargs cargo test".format(sys.argv[0]))
82+
print("python3 {} 3 1 -p | xargs cargo test".format(sys.argv[0]))
83+
print("python3 {} 3 2 -p | xargs cargo test".format(sys.argv[0]))
84+
print("---")
85+
sys.exit(1)
86+
87+
if len(sys.argv) < 3 or len(sys.argv) > 4:
88+
print("Usage: python3 {} <total_partitions> <partition_index> [prefix]".format(sys.argv[0]))
89+
sys.exit(1)
90+
91+
total_splits = int(sys.argv[1])
92+
partition_index = int(sys.argv[2])
93+
prefix = sys.argv[3] + " " if len(sys.argv) == 4 else ""
94+
95+
if total_splits <= 0 or partition_index < 0 or partition_index >= total_splits:
96+
print("Invalid input.")
97+
sys.exit(1)
98+
99+
partition = partition_workspace(total_splits, partition_index)
100+
partition_with_prefix = [prefix + member for member in partition]
101+
102+
print("{}".format(" ".join(partition_with_prefix)))

0 commit comments

Comments
 (0)