Commit 794d346
committed
Merge dashpay#6604: ci: extract "slim" container to run functional tests and add
76f99cd doc: update container documentation (Kittywhiskers Van Gogh)
b8b9bb4 ci: split out `multiprocess` linting to a separate step on Actions (Kittywhiskers Van Gogh)
94dedba revert: tentatively drop multiprocess and tsan functional tests (Kittywhiskers Van Gogh)
b379708 trivial: `lldbUpdAltArgs` -> `llvmUpdAltArgs` (Kittywhiskers Van Gogh)
90710b3 ci: avoid making copies when collecting logs, misc. script fixes (Kittywhiskers Van Gogh)
89d977d ci: use slim container for functional tests (Kittywhiskers Van Gogh)
22e9acd ci: remove old dependencies, packages not used in building or testing (Kittywhiskers Van Gogh)
630bb0c ci: consolidate package installation in `ci` container (Kittywhiskers Van Gogh)
837d10a ci: extract minimum environment needed to run functional tests (Kittywhiskers Van Gogh)
4bcd093 ci: stop using LLVM setup script to setup repo (Kittywhiskers Van Gogh)
a4ce15b ci: rename `ci/Dockerfile`, switch to `devthefuture/dockerfile-x` (Kittywhiskers Van Gogh)
5413955 ci: move packages needed to run `dash-qt` into `develop` container (Kittywhiskers Van Gogh)
Pull request description:
## Motivation
[dash#6583](dashpay#6583) introduced functional tests to GitHub Actions and due to a combination of limited disk space available on runners and restrictions on ways to free up said disk space (as described in [dash#6583](dashpay#6583)), `multiprocess` and `tsan` builds had to be excluded ([commit](dashpay@3461c14)).
Despite measures to reign in disk usage ([source](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/ci/dash/slim-workspace.sh)), we are experiencing sporadic failures due to space exhaustion ([build](https://github.com/dashpay/dash/actions/runs/13572578591/job/37941577942), [build](https://github.com/dashpay/dash/actions/runs/13572201698/job/37940403595)). To take care of this, we are going to extract portions of our `ci` container into `ci-slim`, containing only the parts needed to run (sanitizer-enabled) binaries, functional tests and (some) linters.
## Additional Information
* We have used [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus) to allow container definitions inherit other container definitions, which has allowed for a ready-to-use interactive container ([`develop`](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/contrib/containers/develop/Dockerfile)) to be built on top of our CI container ([`ci`](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/contrib/containers/ci/Dockerfile)) with ease (introduced in [dash#4336](dashpay#4336)).
Unfortunately, since then, there has been little activity at [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus), last code contribution was ~4 years ago ([commit](edrevo/dockerfile-plus@90dc08d)) and as this PR splits off `ci` to `ci-slim`, it appears `dockerfile-plus` has difficulty in working with `Dockerfiles` that inherit other `Dockerfiles` (see below).
<details>
<summary>Error message:</summary>
```
$ docker-compose build
#0 building with "default" instance using docker driver
[...]
#1 [container internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.57kB done
#1 DONE 0.0s
[...]
#7 [container internal] load build definition from Dockerfile
#7 DONE 0.0s
failed to solve: failed to create LLB definition: dockerfile parse error line 9: unknown instruction: INCLUDE+
```
</details>
To mitigate this, we are switching over to [`devthefuture/dockerfile-x`](https://codeberg.org/devthefuture/dockerfile-x), which come with minor changes in syntax.
* As [`devthefuture/dockerfile-x`](https://codeberg.org/devthefuture/dockerfile-x) (like [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus)) relies on BuildKit, it has been explicitly enabled for GitLab, without which, it fails ([build](https://gitlab.com/dashpay/dash/-/jobs/9276738177#L194)).
* As scoping in `Dockerfile`s are relative to the `Dockerfile` itself ([source](https://codeberg.org/devthefuture/dockerfile-x#scoping)), `develop/Dockerfile` can include `ci/Dockerfile` by setting the context one directory behind (which is possible for the root `Dockerfile`) but when `ci/Dockerfile` is parsed, it will search within the `ci` folder and not from the context folder set earlier and so, including `ci-slim/Dockerfile` would come up empty as it would resolve to `ci/ci-slim/Dockerfile`, which doesn't exist.
To work around this, both `ci` and `ci-slim` definitions need to share the same directory and to allow for that, `ci/Dockerfile` is now `ci/ci.Dockerfile`, so that `develop/Dockerfile` can set the context a directory behind, find `ci/ci.Dockerfile`, which will search within `ci` for `./ci-slim.Dockerfile`, which will be found successfully.
* We have switched away from using the official LLVM installation script to importing the repository manually as we need only a select few packages to run sanitizer-enabled builds and functional tests and the script only gives you a choice between default packages and all packages.
* While avoiding LLVM packages entirely in the `ci-slim` container was most desirable, doing so breaks sanitizer builds ([build](https://github.com/kwvg/dash/actions/runs/13585208794/job/37981496193#step:6:2571))
* Switching over to manual import also let us drop a few packages as the script has to probe and work around bugs while our base environment remains relatively similar.
* Space occupied by containers.
Calculated by modifying `develop/docker-compose.yml` to point at different `Dockerfile`s and building them with `docker compose`, then passing them to [dive](https://github.com/wagoodman/dive) v0.12.0.
| Name | Commit | Size | Efficiency Score | Potential Waste |
| --------- | -------------------- | ------ | ---------------- | --------------- |
| `ci` | `develop` (26ea618) | 4.4 GB | 99% | 74 MB |
| `develop` | `develop` (26ea618) | 4.5 GB | 99% | 78 MB |
| `ci-slim` | This PR (db6a3fd0) | 1.6 GB | 97% | 64 MB |
| `ci` | This PR (db6a3fd0) | 4.3 GB | 99% | 70 MB |
| `develop` | This PR (db6a3fd0) | 4.3 GB | 99% | 74 MB |
## Checklist
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK 76f99cd
PastaPastaPasta:
utACK 76f99cd
Tree-SHA512: c0afc570dfc9f6db3dbacbf40b4a2416e993b5e2e503694b9b564a4e00600aa4c290b7c7395f0abc1a7b269c488ed86f4c4c2e64436f38b7ea1653057db83259multiprocess and tsan, drop unneeded packagesFile tree
15 files changed
+354
-272
lines changed- .github/workflows
- ci/dash
- contrib/containers
- ci
- develop
15 files changed
+354
-272
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
5 | 18 | | |
6 | 19 | | |
7 | 20 | | |
8 | | - | |
| 21 | + | |
9 | 22 | | |
10 | 23 | | |
11 | 24 | | |
| |||
44 | 57 | | |
45 | 58 | | |
46 | 59 | | |
47 | | - | |
48 | | - | |
| 60 | + | |
| 61 | + | |
49 | 62 | | |
50 | 63 | | |
51 | | - | |
52 | | - | |
53 | | - | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
54 | 67 | | |
55 | | - | |
56 | | - | |
57 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
58 | 71 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | | - | |
76 | | - | |
| 75 | + | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
90 | 98 | | |
91 | 99 | | |
92 | 100 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
19 | 31 | | |
20 | 32 | | |
21 | 33 | | |
| |||
158 | 170 | | |
159 | 171 | | |
160 | 172 | | |
161 | | - | |
| 173 | + | |
162 | 174 | | |
163 | 175 | | |
164 | 176 | | |
165 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
166 | 187 | | |
167 | 188 | | |
168 | 189 | | |
169 | 190 | | |
170 | | - | |
| 191 | + | |
171 | 192 | | |
172 | 193 | | |
173 | 194 | | |
174 | | - | |
| 195 | + | |
175 | 196 | | |
176 | 197 | | |
177 | 198 | | |
178 | 199 | | |
179 | | - | |
| 200 | + | |
180 | 201 | | |
181 | 202 | | |
182 | 203 | | |
183 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
184 | 214 | | |
185 | 215 | | |
186 | 216 | | |
187 | 217 | | |
188 | | - | |
| 218 | + | |
189 | 219 | | |
190 | 220 | | |
191 | 221 | | |
192 | | - | |
| 222 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | | - | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
59 | | - | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
| |||
116 | 117 | | |
117 | 118 | | |
118 | 119 | | |
119 | | - | |
| 120 | + | |
120 | 121 | | |
121 | 122 | | |
122 | 123 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
15 | | - | |
| 14 | + | |
16 | 15 | | |
17 | 16 | | |
18 | 17 | | |
| |||
51 | 50 | | |
52 | 51 | | |
53 | 52 | | |
| 53 | + | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| |||
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
84 | 73 | | |
85 | 74 | | |
86 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
| 69 | + | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
| |||
0 commit comments