Skip to content

refactor(store): consumers reach the store only through its interface contract - #490

Draft
eribertocz wants to merge 21 commits into
refactor/486-contract-foundationsfrom
refactor/486-store-contract
Draft

eribertocz wants to merge 21 commits into
refactor/486-contract-foundationsfrom
refactor/486-store-contract

Conversation

@eribertocz

@eribertocz eribertocz commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Part of #486 — step 2 (store); stacked on step 1, retarget to main after it merges.

Draft until local full and CI are green. Rebased on #489 (20f1330): aware datetimes cross, so the contract is declared and its kit runs.

Size

About 5,000 counted lines, not split: #486 §5 makes step 2 one pull request for the store's contract, its kit, its api and every consumer moved onto it, because a consumer half-moved would import both the contract and the internals it replaces. Most lines are the mechanical move of ~75 consumer files and the kit's 153 cases.

Files and their purposes

File Purpose
packages/contracts/src/orvex/contracts/store/interface.py The store's interface contract: 44 operations by 8 capabilities, their codes, the ClickHouse required settings
…/store/errors.py (moved from orvex/store/errors.py) The contract's error type StoreError, its codes, redaction and builders (source_not_found, separation_refused, SNAPSHOT_CHANGED, FILE_REFUSED)
…/store/values.py Values the contract carries besides the generated types: limits, targets, columns, handles, Held, JsonFrame
…/store/kit.py The conformance kit: 153 cases, no skips, security cases, one idempotency case per state-changing operation
…/store/kit_codes.py One case per (operation, declared code); the kit is refused when a declared code has no case (KIT_CODE_UNPROVEN)
…/store/kit_services.py The ClickHouse server some cases need, from the environment; missing, the case fails
packages/testing/ (orvex-testing) Test-only distribution of the shared helpers (bounds, environment, fixtures, facade_interface, azurite), never shipped
…/store/kit_support.py What the kit's cases are built from (lakes, statements, reads, refusals, per-run scenarios)
…/store/client.py The consumer's side: StoreClient, Lake, Transaction, ReadHandle, every call through a binding with its envelope
…/store/client_sources.py The consumer's side of existing lakes and ClickHouse
packages/store/src/orvex/store/api/__init__.py The store's one public face: Store
…/store/api/service.py Store, the implementation of the contract
…/store/api/held.py What the implementation holds open by handle, and each state-changing outcome by idempotency key
packages/store/src/orvex/store/module.toml The store's manifest
packages/store/benchmarks/source_stream.py (moved from tools/benchmarks/) The store's own benchmark of its internals
packages/server/src/orvex/server/store_access.py How the server reaches the store: a client on the assembly's provider
packages/server/src/orvex/server/assembly.py + provider: the chosen implementation, built once per process
70+ consumer files in server, conformance/runners, tools, tests Moved from orvex.store.* onto the contract
packages/store/tests/test_contract_kit.py, contract_client.py, test_clickhouse_scanner_fuzzing.py, parquet_footer.py (moved) The kit run, its plants, the store's tests through its client, the scanner fuzzing moved from tests/properties
packages/server/tests/store_faults.py Faults the server's tests inject at the store binding, never in store internals

Laws touched

Law 7 (limits): unchanged, the kit refuses a lake without every limit. Exact values: the kit proves DECIMAL(38,10), HUGEINT, TIMESTAMPTZ and NULL through both bindings. Fail closed: forged or closed handles are refused (STORE_PARAMETER_INVALID); undeclared errors never cross.

Case matrix

Case Expected Test
Every operation, in process and serialized whole kit passes, 58 cases, same outcomes test_contract_kit.py::test_the_store_passes_its_whole_kit_in_process_and_serialized
Planted: execute ignores its key execute.idempotent fails test_the_kit_fails_a_planted_broken_guarantee
Planted: DECIMAL read as float start-read.exact fails same
Planted: forged read handle answered read-batch.ends fails same
Manifest provides every capability from api.Store, requires none test_the_manifest_provides_every_capability_from_the_api_package
Overlap refused before attach no ATTACH test_existing_lake_files.py::test_an_overlapping_declaration_is_refused_before_anything_is_attached
Server faults (other writer, conflict, kill, Ctrl-C, time limit) records as before, injected at the binding packages/server/tests/test_run_records.py
Consumer imports of orvex.store 0 store violations python -m tools.checks.boundaries

Evidence (before the rebase onto the updated #489)

  • python -m tools.checks.boundaries: boundaries: 0 finding(s); consumer report: no store finding.
  • mypy (contracts, store, server): no issues. Kit: 58/58 cases and 644 envelope cases pass, both bindings.
  • Touched tests: 1040 passed, 4 failed, all 4 then fixed (23/23 run-record tests pass).
  • Not run: verify fast (timed out on the loaded machine), local full, CI.

Error codes

Each operation declares only the codes it raises (interface.ERRORS_OF, 110 pairs) and the kit proves every one. Pending: STORE_EXTENSION_UNAVAILABLE and ClickHouse ENGINE_VERSION_UNTESTED need the per-case process environments being added to the foundations; until then they are not declared (a raise of either is refused as INTERFACE_ERROR_UNDECLARED), and the check has no exemption for them. Defense-in-depth codes no environment can provoke without patching the store are not declared either: existing-lake SOURCE_NOT_READ_ONLY (the store sets the read-only session itself) and STORE_CATALOG_UNAVAILABLE (object-storage secret refused).

Open, for the Owner

  1. Shared test helpers (tests/support: bounds, environment, fixtures, facade_interface, azurite) are also used by server and repository tests: moving them would duplicate them. A shared home needs a decision; 6 store test imports remain, so the standalone store job still fails on them.
  2. The envelope's project_id and revision are the server's own (instance, current) and the correlation id is new per client: carrying the request's real ones needs every server entry point to pass a caller.
  3. Every operation declares every store code; the foundations now compare per-operation codes, so they should be narrowed per operation.
  4. Repository tests, runners and tools reach the store through orvex.server.store_access, a server import.

Shared files changed (append-only)

tools/ci/checks.toml (standalone-store job, regressions entry), tools/checks/materialization-allowlist.toml (kit entries), .github/workflows/interfaces.yml (standalone-store job), .github/workflows/property-nightly.yml (the moved fuzzing test), pyproject.toml (T20 for module benchmarks).


Generated by Claude Code

Claude Code and others added 7 commits November 15, 2026 00:24
The format, kits, bindings and manifest; the assembly check; the consumer
boundaries; contract compatibility; the standalone job. Written before the
code: every test fails without it (#486 step 1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Step 1 of #486 (foundations): the interface-contract format, conformance
kits with a recorded count and no skips, the in-process and serializing
bindings, the module manifest, the assembly capability check at server
start, the consumer boundaries in report mode, the compatibility check
against the last release, the standalone job on orvex-contracts, and
ADR-0033 (Proposed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ncy and adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…est adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…it exists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…requires them

tools/ci/protection.py compares the required checks with main's live rulesets, so a
check becomes required only after the Owner adds it there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… codec and assembly

- only a binding creates a refusal; anything else an implementation raises is undeclared, and
  the kit counts no refusal found down a cause chain;
- compatibility compares each operation's error codes and default values;
- one import scanner serves the module and consumer rules and the dynamic-import note;
- models cross field by field, Arrow tables as IPC (pyarrow declared);
- assembly chooses among several implementations by configuration and checks construction,
  signatures (envelope included) and the operations of the provided version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz eribertocz added owner-merge Touches governed paths; only the Owner merges type:refactor Internal restructuring without behavior change module:store module:server module:contracts risk:A labels Nov 15, 2026 — with Claude
Claude Code and others added 4 commits November 15, 2026 08:11
…break it

select.select refuses a descriptor at or past FD_SETSIZE; CI's test part 9 reached it and the
ctrl-c watcher thread raised. The scheduler's watcher had the same defect: both now use a
selector, and each has a regression test with 1,100 open files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om one call context

- AwareDatetime and constrained types are value types; IANA zones, datetime.timezone and
  pydantic's TzInfo each come back as themselves; a naive datetime is refused; every generated
  store type may now cross (pydantic-core declared);
- CallContext holds the correlation id, actor, project and revision, refuses missing or
  placeholder values, and makes every envelope; the server builds one correlation id per HTTP
  request (a well-formed X-Request-Id is kept), CLI command or scheduled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ependency of orvex-contracts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om its declared environment

A case declares variables and an extension directory (copied with or without given
extensions); the kit runs it in a child process built from that declaration alone and reads
its result back through the serializing codec, so no kit patches an implementation's internals
and production holds no switch for a test. One case's run moves to kit_case.py, shared by the
kit and the child. Plant: a case whose declared environment is ignored fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz
eribertocz force-pushed the refactor/486-store-contract branch from 9f69d60 to b874e15 Compare November 15, 2026 12:49
Claude Code and others added 9 commits November 15, 2026 09:11
…-testing distribution

`bounds`, `environment`, `fixtures`, `facade_interface` and `azurite` move from `tests/support`
to `packages/testing` (`orvex.testing`), never shipped and depending on no module, and every
test imports them from there. The boundaries check refuses them in product code and the consumer
rule allows them in tests only; the product-name check knows the `orvex-testing` package; the
standalone job installs its wheel beside a module whose tests declare it; the Windows job
installs it for the tests it runs.

Part of #486.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…s consumers

The store's kit run in process and serialized with planted broken guarantees, the Arrow IPC
transport, the codec's aliased models and fixed offsets, the ClickHouse scanner fuzzing as the
store's own test, the refusal before attach, and the store's consumers' tests on the contract.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… contract

`orvex.contracts.store`: the interface contract (every operation consumers use, by capability),
its kit (58 cases, no skips), its errors with their codes and builders, its values and handles,
its Arrow IPC transport and its consumer client. `orvex.store.api.Store` implements it, with
`module.toml`. `server`, the runners, `tools` and the repository tests call the contract; the
boundaries report shows no store violation. The standalone store job is added.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
The time limit, the conflict and the other-writer faults of the run-record tests are injected
through the store contract's binding (a helper's own calls pass the interception), and the
canonical-bytes subprocess reaches the encoder through the store client.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… Arrow table type

The store contract's reads, loads and JSON encoding carry `pyarrow.Table` values, encoded by
the foundations' codec; the store's own IPC module and its codec changes are dropped, and the
standalone store job is optional like the other interface jobs until main requires them.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ct's files are counted

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ugh its composition root

The consumer rule allows `server.assembly`, and nothing else of the server, outside the server.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…e kit proves every one

The store contract's operations declare their own error codes; `kit_codes` holds one case per
(operation, code), each making that operation refuse with that code, and the kit is refused
when a declared code has no case or a case proves an undeclared code (KIT_CODE_UNPROVEN). The
ClickHouse cases take a server from the environment and fail without one (`kit_services`);
refusal-only cases the code cases now prove are removed from the kit (153 cases).

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… extras, CI installs the helpers

The structure check's store façade becomes `orvex.store.api`, which the implementation's own
modules may not import back; a test dependency may name its extras (`moto[server]`), compared by
its name; the Windows job installs `orvex-testing`; the store tests import the contract's types
from the contract; and the test helpers' docstring names no product.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz
eribertocz force-pushed the refactor/486-store-contract branch from f778a76 to 5fbab76 Compare November 15, 2026 13:12
…ven in a process of their own

`open_lake` and `open_existing_lake` declare STORE_EXTENSION_UNAVAILABLE, proven with an empty
extension directory and a proxy nothing answers on; `connect_clickhouse` declares
ENGINE_VERSION_UNTESTED, proven against a ClickHouse 26.9 service the kit's CI jobs start. Each
case declares its environment and runs in its own process (the foundations' kit_process), so
no case patches the store; their checks live beside the kit, where that process finds them.

Part of #486 (step 2, store).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz
eribertocz force-pushed the refactor/486-contract-foundations branch from 6ba558b to 94bf619 Compare November 15, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:contracts module:server module:store owner-merge Touches governed paths; only the Owner merges risk:A type:refactor Internal restructuring without behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant