Repository navigation
Commit 9f7d5ea
ci(migrations): Cache the migrated test databases between runs (#128394)
`backend / backend migration tests` spends about 237s on master CI
building `test_region`, `test_control` and `test_secondary`. The first
test's setup runs every migration from scratch.
With this change:
- On master, after the tests pass, the job saves a `pg_dump` of the
three test databases plus a manifest of the migration files it was built
from. The cache key hashes `src/**/migrations/*.py` and
`src/sentry/new_migrations/**`.
- Every run restores the exact key, or the newest master dump through
`restore-keys`, before running pytest. The job already uses
`--reuse-db`, so pytest-django keeps the restored databases and applies
only the migrations added since the dump.
- If the manifest lists a migration file this checkout doesn't have, the
script skips the restore and pytest builds the databases from scratch as
it does today. That happens when a PR branch is behind the master commit
that saved the dump.
- `.github/file-filters.yml`: changes to the new script now trigger the
migration tests.
**Local measurements (single migration test, `--migrations
--reuse-db`)**
- From scratch: 306s DB setup
- After restoring a dump (~6s): ~40s DB setup; the test passes
- I also checked that a manifest with an unknown migration skips the
restore, and that a missing cache falls through to a normal build.
**Not verified yet**
- No cache exists until this lands, so this PR's own run takes the "no
saved databases" path. The first master run saves the dump, and the run
after that is the first real restore in CI.
- I didn't check the case where the dump is behind and pytest applies
the newer migrations on top. That path is plain pytest-django `keepdb`
plus `migrate`.
<!-- junior-request-attribution:start -->
via @joshuarli.
<!-- junior-request-attribution:end -->
<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3ACUHS29QJ0%3A1792602543.346169 -->
--
[View Junior
Session](https://junior-prod.sentry.dev/conversations/slack%3ACUHS29QJ0%3A1792602543.346169)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3ACUHS29QJ0%3A1792602543.346169/?project=4510944073809921)
<!-- junior-session-footer:end -->
---------
Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>1 parent bda7d4d commit 9f7d5ea
3 files changed
Lines changed: 99 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
175 | 175 | | |
176 | 176 | | |
177 | 177 | | |
| 178 | + | |
178 | 179 | | |
179 | 180 | | |
180 | 181 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
369 | 369 | | |
370 | 370 | | |
371 | 371 | | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
372 | 375 | | |
373 | 376 | | |
374 | 377 | | |
| |||
379 | 382 | | |
380 | 383 | | |
381 | 384 | | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
382 | 397 | | |
383 | 398 | | |
384 | 399 | | |
385 | 400 | | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
386 | 412 | | |
387 | 413 | | |
388 | 414 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
0 commit comments