Skip to content

refactor(transform): transform behind its interface contract (#486 step 3) - #495

Draft
eribertocz wants to merge 16 commits into
refactor/486-contract-foundationsfrom
refactor/486-transform-contract
Draft

eribertocz wants to merge 16 commits into
refactor/486-contract-foundationsfrom
refactor/486-transform-contract

Conversation

@eribertocz

Copy link
Copy Markdown
Owner

Part of #486 — step 3 (transform); stacked on step 1, retarget to main after it merges.

What changed

transform now has one public face, orvex.transform.api.Transform, which implements the new interface contract orvex.contracts.transform (version 1.0). Every consumer reaches transform only through that contract, with a complete call envelope.

  • Contract. 33 operations in six capabilities (models, compile, run, records, schedule, lineage). These are all the operations the server used. Every input, output and error is a plain value, so the contract can later cross a process boundary (feat(server): start server mode and supervise its transformation worker #156, Orvex Flow). Transform never opens the lake. The consumer reads the store, gives transform what it read and executes the requests transform answers with. The three callbacks transform used to receive (since-bound proof, automatic filter, column lineage) became a data exchange: an operation answers ColumnsNeeded with the relations it could not find, and the consumer describes them and calls again (ColumnsExchange). An implementation that asks again for a relation it was given fails closed (TRANSFORM_COLUMNS_REPEATED).
  • Value types moved, not copied. Model, Discovery, Graph, the run steps (Check, Count, Refuse, Watermark, Land, ModelTest, Reconcile, Comparison), History, Plan, the filter Refusal and the lineage scope are now defined once, in the contract. Transform's internals import them from there. There is no re-export and no second definition. frozenset and Mapping fields became tuple and dict, so the values can be serialized.
  • Errors. Transform's only shared exception class was the configuration-file reader's ConfigProblem, which the server also used for project.yaml and connections.yaml. The reader (config_file.py) and its error moved unchanged to orvex.contracts.config_file, with the same attributes and kinds. TransformError is the contract's error type. No operation declares a code, because a model's problems are ModelError values and are never raised. So anything an implementation raises reaches a consumer as INTERFACE_ERROR_UNDECLARED (fail closed).
  • No state-changing operation. Transform never writes: it answers with statements that the server executes in its own transaction. So no operation carries an idempotency key, and the kit has no idempotency case. If a state-changing operation is ever added, Kit refuses it until it gets one (KIT_IDEMPOTENCY_UNCOVERED).
  • Kit. 37 cases with no skip list, recorded in interface-record.json. They run in process and serialized, by consumers at N-1 and N, plus the envelope cases.
  • Server. transform_calls.transform() is the contract's protocol (Transform), served through the in-process binding. Calls are therefore type-checked against the contract (mypy found 20 call mismatches, all fixed). Each call carries envelope("<operation>"), made by the foundations' CallContext of the command (command_caller.project_context). The context holds real values only: the actor is the operating-system user, the project is its folder, and the revision is git:<commit>, or git:<commit>+files:<sha256> when the project files differ from that commit, or files:<sha256> without Git. One calling(folder) block keeps one context, and so one correlation id, across a command, for example a scheduler cycle or a check followed by its run. Outside a block no envelope can be made (CALL_CONTEXT_MISSING). assembly.implementation_of is the one place that builds the chosen implementation. Leftover-table logic, which names transform's own watermarks table, moved into transform (leftovers.py).
  • Standalone job. python -m tools.ci.standalone transform builds the wheel from the sdist and installs it outside the checkout with only orvex-contracts. It then runs transform's own tests, including the kit. Transform requires no interface contract, so no test adapter is needed. Its manifest lists no requires, and it runs without semantic, the UI, BI or AI.

Files and their purposes

File Purpose
packages/contracts/src/orvex/contracts/transform/__init__.py the transform contract package
.../transform/interface.py the operations (Transform protocol), TransformError and TRANSFORM
.../transform/models.py model, discovery and graph values
.../transform/steps.py run-step values and $since
.../transform/scheduling.py history, plan and scheduler intervals
.../transform/analysis.py the columns exchange values, filter refusal, reconciliation staging, lineage inputs
.../transform/columns.py the consumer's half of the columns exchange
.../transform/kit.py the kit (37 cases, recorded)
.../transform/kit_project.py the synthetic project the kit runs on
.../transform/kit_models.py, kit_runs.py, kit_schedule.py the kit's cases, by capability
packages/contracts/src/orvex/contracts/config_file.py moved from transform: the safe YAML configuration reader shared by project and model files
packages/transform/src/orvex/transform/api.py the one implementation of the contract
packages/transform/src/orvex/transform/leftovers.py moved from server: leftover tables and their drop statements
packages/transform/src/orvex/transform/module.toml transform's manifest
packages/server/src/orvex/server/transform_calls.py the server's calls to the transform contract, with their envelope
packages/server/src/orvex/server/command_caller.py the CallContext of a CLI command or scheduled run: user, folder, revision
packages/server/src/orvex/server/project_files.py the files of a project a check reads (shared by the scheduler's fingerprint and the revision digest)
packages/server/src/orvex/server/assembly.py + implementation_of
packages/server/tests/record_statements.py test double: the run's contract with record_statements replaced
server history, lake_open, leftovers, lineage, project, project_check, run, run_metadata, run_reads, run_steps, scheduler, validate moved onto the contract (no orvex.transform import left)
transform internals (auto_filter, graph, incremental*, lineage, model_tests, models, reconciliation, schedule) import the contract's value types instead of defining them
tools/checks/module_dependencies.py tzdata counts as used where zoneinfo is imported and it is declared
tools/checks/data_readers.py also covers contracts, now that the configuration reader lives there

Case matrix

Criterion / rejection case Test Result
The whole kit passes in process and serialized, N-1 and N, with no skip; the count is recorded (37) packages/transform/tests/test_api_kit.py::test_the_whole_kit_passes_in_process_and_serialized pass
Every contract operation is implemented by api.Transform test_api_kit.py::test_every_operation_is_implemented pass
Planted: an implementation that drops $since's time zone fails the kit test_api_kit.py::test_a_planted_wrong_implementation_fails_the_kit[_Floated-…] pass (KIT_CASE_FAILED: since-value.lookback)
Planted: an operation that returns a live object fails the serialized run test_api_kit.py::…[_Live-…] pass (KIT_CASE_FAILED: lineage-project-key.slug)
Exact values, NULLs and time zones (38-digit counts, DECIMAL(38,4), floats never compared, NULL watermark not stored, DST in America/New_York) kit cases record-statements.exact, measure-difference.exact, watermark-statements.exact, since-value.lookback, next-due.time-zone, schedule-plan.never-run-due pass
No envelope is made, and so no call is sent, outside a command packages/server/tests/test_transform_calls.py::test_no_envelope_is_made_outside_a_command pass
A call carries the command's real context; one command keeps one context per project test_transform_calls.py::test_a_call_inside_a_command_carries_its_real_context, ::test_one_command_keeps_one_context_per_project pass
The revision names the files: the commit, the commit plus a digest when they differ, or the digest; an actor that cannot be found is refused packages/server/tests/test_command_caller.py (4 cases) pass
Only contract operations are reachable test_transform_calls.py::test_only_the_contract_operations_can_be_called pass
The columns exchange describes each relation once; asking again or for nothing fails closed packages/contracts/tests/test_transform_columns.py (3 cases) pass
Assembly builds the one installed implementation; a contract with none is refused packages/server/tests/test_assembly.py::test_the_server_calls_the_one_installed_transform_implementation, ::test_a_contract_no_installed_module_provides_is_refused pass
No consumer imports a transform internal (boundaries report: zero transform findings) tests/tools/test_consumer_boundaries.py::test_no_consumer_uses_a_transform_internal pass
Planted undeclared dependency / unused tzdata without zoneinfo fail the standalone check tests/tools/test_standalone.py::test_tzdata_is_used_by_zoneinfo_only_when_declared and the existing plants pass
Transform installs alone outside the repository and passes its tests and kit python -m tools.ci.standalone transform (CI job standalone) see CI
Behaviour unchanged for every consumer the existing server suite (check, run, schedule, runs, lineage, records, source models) pass locally

How it was verified

  • verify fast: see the comment that follows (the machine is shared and loaded; full runs in CI first, then once locally).
  • Locally: pytest packages/server/tests packages/contracts/tests packages/transform/tests tests/tools/test_standalone.py tests/tools/test_consumer_boundaries.py. The six failures it found were all fixed at their cause: tests that patched run.record_statements now replace it at the contract. The re-run passes.
  • mypy -p orvex.transform -p orvex.server -p orvex.contracts and the changed tests: clean. tools.contracts.interface_record --check: current. tools.ci.checks: matches the workflows.

Size

Over the guide. The change is one coherent move, done per file and mechanically: types moved into the contract, call sites rewritten to the typed protocol, and the 12 server consumer files moved. The kit and the contract are new code. Review the contract (interface.py, analysis.py) and api.py first, then transform_calls.py and project_check.py.

Time and CI-first errors

Filled in when CI finishes.

Not done here

  • server still names _meta.run_records_v2 (run_reads._committed_snapshot), a transform table, as a string. The G2 storage-name check is not part of this step.
  • The HTTP paths make no transform call, so request_context is not used here.

Laws touched

Law 11 (exact values): unchanged and proven by the kit (exact counts, DECIMAL, floats refused). The fail-closed rule applies to every refused envelope.

Contracts touched

transform interface contract 1.0: new. orvex.contracts.config_file: moved from transform without change.

Dependencies added

pyyaml 6.0.3 becomes a dependency of orvex-contracts. It was already transform's, at the same pin and license, and it follows the configuration reader that moved. transform no longer declares pydantic or pyyaml, because it does not import them.

Shared files changed

tools/ci/checks.toml (appended standalone-transform; the standalone job digest updated), .github/workflows/interfaces.yml (appended one step), tools/checks/data-reader-allowlist.toml (paths of the moved reader), tools/ci/test-durations.json (path of the moved test), packages/contracts/interface-record.json (regenerated).

Risks

  • The standalone job digest will conflict with step 2 (store), which appends to the same job. Resolve it by recomputing the digest.
  • When the foundations gain the per-operation error codes and the Arrow codec, the record is regenerated on rebase.

Generated by Claude Code

Claude Code and others added 16 commits November 15, 2026 00:24
The format, kits, bindings and manifest; the assembly check; the consumer
boundaries; contract compatibility; the standalone job. Written before the
code: every test fails without it (#486 step 1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Step 1 of #486 (foundations): the interface-contract format, conformance
kits with a recorded count and no skips, the in-process and serializing
bindings, the module manifest, the assembly capability check at server
start, the consumer boundaries in report mode, the compatibility check
against the last release, the standalone job on orvex-contracts, and
ADR-0033 (Proposed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ncy and adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…est adapters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…it exists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…requires them

tools/ci/protection.py compares the required checks with main's live rulesets, so a
check becomes required only after the Owner adds it there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… codec and assembly

- only a binding creates a refusal; anything else an implementation raises is undeclared, and
  the kit counts no refusal found down a cause chain;
- compatibility compares each operation's error codes and default values;
- one import scanner serves the module and consumer rules and the dynamic-import note;
- models cross field by field, Arrow tables as IPC (pyarrow declared);
- assembly chooses among several implementations by configuration and checks construction,
  signatures (envelope included) and the operations of the provided version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…break it

select.select refuses a descriptor at or past FD_SETSIZE; CI's test part 9 reached it and the
ctrl-c watcher thread raised. The scheduler's watcher had the same defect: both now use a
selector, and each has a regression test with 1,100 open files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om one call context

- AwareDatetime and constrained types are value types; IANA zones, datetime.timezone and
  pydantic's TzInfo each come back as themselves; a naive datetime is refused; every generated
  store type may now cross (pydantic-core declared);
- CallContext holds the correlation id, actor, project and revision, refuses missing or
  placeholder values, and makes every envelope; the server builds one correlation id per HTTP
  request (a well-formed X-Request-Id is kept), CLI command or scheduled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ependency of orvex-contracts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om its declared environment

A case declares variables and an extension directory (copied with or without given
extensions); the kit runs it in a child process built from that declaration alone and reads
its result back through the serializing codec, so no kit patches an implementation's internals
and production holds no switch for a test. One case's run moves to kit_case.py, shared by the
kit and the child. Plant: a case whose declared environment is ignored fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
The kit passes in process and serialized, planted wrong implementations
fail it, server calls carry an envelope and fail closed without one, the
columns exchange never loops, assembly gives the one implementation, no
consumer imports a transform internal, and tzdata counts as used through
zoneinfo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
orvex.contracts.transform declares the operations every consumer uses,
their value types and a 37-case kit; orvex.transform.api implements it;
packages/transform/module.toml provides it. The server reaches transform
only through the contract, with an envelope on every call. The shared
configuration-file reader and its errors move to orvex.contracts. The
standalone job builds, installs and tests transform alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Calls to transform name the project by its folder, and a block inside
another for the same project keeps its correlation id, so a scheduler
cycle or a run is followed under one id. Tests that replace the run's
record statements do it at the contract, through one shared double.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… per-operation format

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Each command builds one CallContext for the project it acts on: the
operating-system user, the project folder and its revision (its Git
commit, with a digest of its files when they differ from it, or the
digest alone without Git). Envelopes come only from that context; none
is made outside a command. The files a check reads are listed once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
@eribertocz eribertocz added module:contracts module:server module:transform owner-merge Touches governed paths; only the Owner merges risk:A type:refactor Internal restructuring without behavior change labels Nov 15, 2026 — with Claude
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:contracts module:server module:transform owner-merge Touches governed paths; only the Owner merges risk:A type:refactor Internal restructuring without behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant