Skip to content

build(repo): latest stable dependencies, coverage floors and supply-chain checks - #465

Merged
eribertocz merged 10 commits into
mainfrom
build/429-latest-stable-and-coverage
Nov 10, 2026
Merged

eribertocz merged 10 commits into
mainfrom
build/429-latest-stable-and-coverage

Conversation

@eribertocz

@eribertocz eribertocz commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Closes #429. Risk tier A (lock files, CI workflows, store engine versions). Label owner-merge: it changes .github/workflows/, docs/process/git-workflow.md and two new governed files.

What changes, by scope item

  1. Python, measured on PyPI on 2026-11-10.
    • Updated: fastapi 0.162.3, pyarrow 26.0.1, pydantic 2.14.3, datamodel-code-generator 0.92.0, hypothesis 6.170.15, pytest 9.2.0, ruff 0.17.4, types-pyyaml 6.0.12.20261103.
    • Both lock files were regenerated.
    • The generated contract clients are unchanged: 0 stale files under the new generator.
  2. Web, measured on npm on 2026-11-10.
    • Updated: @sveltejs/kit 3.2.0, @tailwindcss/vite and tailwindcss 4.3.5, eslint-plugin-svelte 3.24.1, globals 17.14.0, jsdom 31.0.0 (a major version), svelte-check 4.7.8, typescript-eslint 8.72.2, vite 8.3.7, vitest 5.1.0.
    • package-lock.json was regenerated.
    • @types/node stays on 24.19.2, the newest of the runtime's line (build(web): move to Node 26 LTS on its promotion date #23).
  3. Re-checked.
    • The Actions pins are each at their newest release; unchanged.
    • DuckDB 2.0.0 is current.
    • pip moves to 26.3 in every workflow.
    • ClickHouse moves to 26.10.2.103, the newest stable. Its tested versions become 26.8 (LTS) and 26.10 (latest stable), as ADR-0022 requires. All ClickHouse conformance and store tests pass on 26.10 here (349 passed).
  4. Coverage of the critical sources (moved from test(repo): gates and tests that cannot fail #427).
    • A new profile, critical_sources in tools/ci/checks.toml, lists the code the critical tests protect: access, policies, compiler, correctness, adapters, exact values, isolation and the SQL guard. That is 39 files.
    • A new plugin, tools.ci.coverage_run, measures branch coverage only when a part runs its share of the whole suite (PYTEST_COVERAGE=1 from select.py).
    • Each part uploads its data, or a not-measured marker.
    • A new job, python-coverage-<python>, combines the parts per Python and runs tools/ci/coverage_floor.py. Each Python must reach every floor, so the lower of the two decides.
    • The check fails on: a fall below the floor, a missing floor, a floor for a module that is not critical, a module that was never instrumented, a missing or mixed part, and empty or unreadable data.
    • A module that is one percent or more above its floor is only reported.
    • python-tests-<python>, which is already required, now needs this job.
  5. Supply chain.
    • Integrity: pip checks the hashes in pylock.toml. python-lock now shows that a tampered hash fails the install, with the new tools/ci/tamper.py.
    • Reproducibility: two clean installs, now on both 3.15 and 3.14, must give the same pip freeze --all.
    • Vulnerabilities: pip-audit runs on both Pythons. The new web-supply-chain job runs npm audit signatures and npm audit.
    • tools/ci/audit.py judges both reports against tools/ci/audit-exceptions.toml. An exception needs its ecosystem, package, advisory ID, reason and review date. An expired, unused or incomplete exception fails too.

Justified exception (rule of #429, tracked by #466)

Package Kept What breaks Tracked by
typescript 6.0.3 (newest 6.x) 7.0.2: svelte-check 4.7.8 accepts only ^5 || ^6, and typescript-eslint 8.72.2 only <6.1.0 #466 (approved by the Owner); Dependabot #320 stays open until then

Coverage floors: measured and reproducible

  • The values are reproducible. Two CI runs with different random test orders printed identical values on 3.15 and 3.14, for all 39 files: run 42884746434 on 08fff6a, then the run on c7b9393.
  • Root cause of the earlier, unstable measurements. Coverage's sysmon core shares sys.monitoring with other tools in the run, which silently dropped its events.
    • On 3.14 the tool was Hypothesis' explain phase: locally, the whole semantic suite's data came out empty.
    • On 3.15 values still varied after that was fixed.
    • Coverage now uses the ctrace core (sys.settrace), which is immune to this. It is about twice as slow, and it runs only on whole-suite runs.
    • While measuring, Hypothesis is also derandomized, with no example database and no explain phase.
  • The floors. Each floor is the lower of the 3.15 and 3.14 values, rounded down; the Reviewer checked all 39 against the run's logs. The floors here differ from earlier commits of this pull request: those floors came from unreproducible measurements and never reached main.
  • One low floor. packages/store/src/orvex/store/source_exactness.py is at 24%. It is a critical module whose branches the suite barely reaches. Its floor records today's value so it cannot fall; raising it is follow-up work.
  • A related test fix. packages/server/tests/test_regular_file.py replaced os.stat for the whole process, so coverage's own callbacks inside the reader tripped it. The trap now fires only on calls the reader makes. The failure was reproduced by running the test alone under coverage.

The ClickHouse image is pinned by digest (sha256:b2bfa4d8…). The digest is read from the CI pull log, because this sandbox cannot reach the Docker registry.

Files and their purpose

File Purpose
pyproject.toml, packages/*/pyproject.toml, pylock.toml, pylock.py314.toml the pins, plus coverage and pip-audit; the coverage settings; the locks
web/package.json, web/package-lock.json the web pins and lock
tools/ci/coverage_run.py (new) the pytest plugin that measures
tools/ci/coverage_floor.py (new) judges the combined data against the floors
tools/ci/coverage-floor.toml (new, governed) the floors
tools/ci/audit.py (new) judges audit reports against the exceptions
tools/ci/audit-exceptions.toml (new, governed) the accepted vulnerabilities (none)
tools/ci/tamper.py (new) the tampered lock copy for the hash check
tools/ci/select.py, tools/ci/checks.toml PYTEST_COVERAGE; the profile, new jobs and digests
.github/workflows/*.yml pip 26.3; coverage; the lock, audit and supply-chain checks; ClickHouse 26.10
packages/store/src/orvex/store/clickhouse.py, tests/conformance/test_clickhouse_engine.py tested ClickHouse versions
docs/process/git-workflow.md, CONTRIBUTING.md the two governed files; pip 26.3
tools/ci/parts.py (new) the number of test parts, standard library only, so the coverage job needs only coverage
contracts/store/v1/README.md behavior change: the tested ClickHouse versions become 26.8 and 26.10, so a 26.9 server is now refused with ENGINE_VERSION_UNTESTED (ADR-0022: the current LTS and the latest stable)

New dependencies, one line each (engineering.md §5):

  • coverage, Apache-2.0: the reference branch-coverage tool. The alternative, pytest-cov, is a wrapper that adds a second dependency.
  • pip-audit, Apache-2.0, maintained by PyPA: the standard Python vulnerability audit. The alternative, osv-scanner, is a Go binary outside the Python toolchain.

Case matrix

Case Test
parts combined; a file never imported is absent, not 0% test_combined_parts_give_each_imported_file_its_branch_coverage
a part's folder or data missing; parts measured and not measured mixed; no part measured passes test_coverage_floor.py (3)
unreadable data; data that measured nothing test_unreadable_or_empty_data_fails, test_data_that_measured_nothing_is_empty_and_fails
below the floor, missing floor, unknown floor, not instrumented; a rise only reported test_each_floor_case_fails_and_a_rise_is_only_reported
the repository floors name exactly the critical sources; floors round down; the command fails closed test_coverage_floor.py (3)
the workflow measures with the plugin, and the coverage result gates python-tests test_the_workflow_measures_the_whole_suite_and_its_result_gates_the_tests
coverage only on the whole suite test_select.py::test_coverage_is_measured_only_on_the_whole_suite
the plugin measures only when asked; Hypothesis is derandomized and never explains while measuring test_coverage_run.py (2)
the descriptor test traps only the reader's own calls (found by the coverage run) test_regular_file.py::test_a_swap_after_the_check_does_not_change_what_is_read
an unmeasured push fails; the command exits 1 below a floor or without a floor; it runs with only coverage installed test_coverage_floor.py (3 new)
an npm finding matched by its GHSA exception; a transitive entry; an exception with no reason refused test_audit.py (2 new)
a vulnerability fails (pip; npm high and critical); a low one passes; an exception passes until its review date; another package's, unused, incomplete or expired exceptions fail; unreadable reports fail test_audit.py (5)
the tampered copy differs by one hash digit; a name pip would not read is refused test_tamper.py (2)
every locked wheel has a hash tests/workspace/test_lock_files.py (already there)

Laws

  • Law 3 (security by design): installs are verified, audited and reproducible.
  • Exactness: ClickHouse 26.10 passes every exactness and parity test.
  • Fail closed: missing data, an unreadable report or an unknown event fails.

Verification

Run locally on Python 3.14:

  • tests/tools, tests/workspace and tests/contracts: 981 passed, 1 not run (root);
  • the ClickHouse conformance and store tests on 26.10.2.103: 349 passed;
  • web: check, lint, 13 tests and build all pass, run on Node 22 (CI uses .nvmrc 24);
  • ruff, mypy (290 files), purpose, structure and tools.ci.checks: clean.

CI runs the whole suite here, because the change touches the locks.

Size

About 900 lines, lock files aside. Half of them are tests.

https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf


Generated by Claude Code

…ate coverage

Python and web dependencies, pip and ClickHouse move to their newest stable
releases (TypeScript stays on 6.0.3: svelte-check and typescript-eslint do
not accept 7 yet). The whole suite measures the critical sources' branch
coverage against governed floors; installs are checked for tampered hashes,
reproducibility on both Pythons, registry signatures and known
vulnerabilities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz eribertocz added module:repo owner-merge Touches governed paths; only the Owner merges risk:A type:build Build system and packaging labels Nov 10, 2026 — with Claude
Claude Code and others added 9 commits November 10, 2026 12:06
The coverage job needs only coverage (the part count moves to a
standard-library module); an unmeasured push fails; the CLI exit codes,
npm exceptions and a missing reason are tested; the tamper check accepts
only a hash refusal; the store contract and the corpus workflow name
ClickHouse 26.10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
The descriptor test replaced os.stat and os.lstat for the whole process,
so coverage and pytest's own reporting tripped it once the suite ran
under coverage. The trap now fires only while the reader is on the stack,
through pathlib included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Coverage's monitoring callbacks run inside the reader and stat its
source the first time a branch fires; reproduced by running the test
alone under the sysmon core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
While measuring, Hypothesis is derandomized and keeps no example
database, so a commit measures the same coverage on every run; the
check prints each module's value for the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
The explain phase registers its own sys.monitoring tool, which drops
coverage's events: a package's data came out empty locally and CI runs
of the same code differed by up to 3%. Without it, two random test orders
measure identical values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
On 3.15 another sys.monitoring tool still dropped sysmon's events, so
the same code measured differently from run to run; the settrace core
is immune to it (identical values across random orders, measured here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz
eribertocz merged commit 761f165 into main Nov 10, 2026
56 checks passed
@eribertocz
eribertocz deleted the build/429-latest-stable-and-coverage branch November 10, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:repo owner-merge Touches governed paths; only the Owner merges risk:A type:build Build system and packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build(repo): dependencies at their latest stable release for P3 (ADR-0022)

1 participant