Repository navigation
refactor(transform): transform behind its interface contract (#486 step 3) - #495
Draft
eribertocz wants to merge 16 commits into
Draft
eribertocz wants to merge 16 commits into
eribertocz wants to merge 16 commits into
Conversation
The format, kits, bindings and manifest; the assembly check; the consumer boundaries; contract compatibility; the standalone job. Written before the code: every test fails without it (#486 step 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Step 1 of #486 (foundations): the interface-contract format, conformance kits with a recorded count and no skips, the in-process and serializing bindings, the module manifest, the assembly capability check at server start, the consumer boundaries in report mode, the compatibility check against the last release, the standalone job on orvex-contracts, and ADR-0033 (Proposed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ncy and adapters Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…est adapters Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…it exists Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…requires them tools/ci/protection.py compares the required checks with main's live rulesets, so a check becomes required only after the Owner adds it there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… codec and assembly - only a binding creates a refusal; anything else an implementation raises is undeclared, and the kit counts no refusal found down a cause chain; - compatibility compares each operation's error codes and default values; - one import scanner serves the module and consumer rules and the dynamic-import note; - models cross field by field, Arrow tables as IPC (pyarrow declared); - assembly chooses among several implementations by configuration and checks construction, signatures (envelope included) and the operations of the provided version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…break it select.select refuses a descriptor at or past FD_SETSIZE; CI's test part 9 reached it and the ctrl-c watcher thread raised. The scheduler's watcher had the same defect: both now use a selector, and each has a regression test with 1,100 open files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om one call context - AwareDatetime and constrained types are value types; IANA zones, datetime.timezone and pydantic's TzInfo each come back as themselves; a naive datetime is refused; every generated store type may now cross (pydantic-core declared); - CallContext holds the correlation id, actor, project and revision, refuses missing or placeholder values, and makes every envelope; the server builds one correlation id per HTTP request (a well-formed X-Request-Id is kept), CLI command or scheduled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…ependency of orvex-contracts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
…om its declared environment A case declares variables and an extension directory (copied with or without given extensions); the kit runs it in a child process built from that declaration alone and reads its result back through the serializing codec, so no kit patches an implementation's internals and production holds no switch for a test. One case's run moves to kit_case.py, shared by the kit and the child. Plant: a case whose declared environment is ignored fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
The kit passes in process and serialized, planted wrong implementations fail it, server calls carry an envelope and fail closed without one, the columns exchange never loops, assembly gives the one implementation, no consumer imports a transform internal, and tzdata counts as used through zoneinfo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
orvex.contracts.transform declares the operations every consumer uses, their value types and a 37-case kit; orvex.transform.api implements it; packages/transform/module.toml provides it. The server reaches transform only through the contract, with an envelope on every call. The shared configuration-file reader and its errors move to orvex.contracts. The standalone job builds, installs and tests transform alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Calls to transform name the project by its folder, and a block inside another for the same project keeps its correlation id, so a scheduler cycle or a run is followed under one id. Tests that replace the run's record statements do it at the contract, through one shared double. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
… per-operation format Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
Each command builds one CallContext for the project it acts on: the operating-system user, the project folder and its revision (its Git commit, with a digest of its files when they differ from it, or the digest alone without Git). Envelopes come only from that context; none is made outside a command. The files a check reads are listed once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFyANPkiUZYq6rvGD2sFLf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #486 — step 3 (transform); stacked on step 1, retarget to main after it merges.
What changed
transformnow has one public face,orvex.transform.api.Transform, which implements the new interface contractorvex.contracts.transform(version 1.0). Every consumer reaches transform only through that contract, with a complete call envelope.models,compile,run,records,schedule,lineage). These are all the operations the server used. Every input, output and error is a plain value, so the contract can later cross a process boundary (feat(server): start server mode and supervise its transformation worker #156, Orvex Flow). Transform never opens the lake. The consumer reads the store, gives transform what it read and executes the requests transform answers with. The three callbacks transform used to receive (since-bound proof, automatic filter, column lineage) became a data exchange: an operation answersColumnsNeededwith the relations it could not find, and the consumer describes them and calls again (ColumnsExchange). An implementation that asks again for a relation it was given fails closed (TRANSFORM_COLUMNS_REPEATED).Model,Discovery,Graph, the run steps (Check,Count,Refuse,Watermark,Land,ModelTest,Reconcile,Comparison),History,Plan, the filterRefusaland the lineage scope are now defined once, in the contract. Transform's internals import them from there. There is no re-export and no second definition.frozensetandMappingfields becametupleanddict, so the values can be serialized.ConfigProblem, which the server also used forproject.yamlandconnections.yaml. The reader (config_file.py) and its error moved unchanged toorvex.contracts.config_file, with the same attributes and kinds.TransformErroris the contract's error type. No operation declares a code, because a model's problems areModelErrorvalues and are never raised. So anything an implementation raises reaches a consumer asINTERFACE_ERROR_UNDECLARED(fail closed).Kitrefuses it until it gets one (KIT_IDEMPOTENCY_UNCOVERED).interface-record.json. They run in process and serialized, by consumers at N-1 and N, plus the envelope cases.transform_calls.transform()is the contract's protocol (Transform), served through the in-process binding. Calls are therefore type-checked against the contract (mypy found 20 call mismatches, all fixed). Each call carriesenvelope("<operation>"), made by the foundations'CallContextof the command (command_caller.project_context). The context holds real values only: the actor is the operating-system user, the project is its folder, and the revision isgit:<commit>, orgit:<commit>+files:<sha256>when the project files differ from that commit, orfiles:<sha256>without Git. Onecalling(folder)block keeps one context, and so one correlation id, across a command, for example a scheduler cycle or a check followed by its run. Outside a block no envelope can be made (CALL_CONTEXT_MISSING).assembly.implementation_ofis the one place that builds the chosen implementation. Leftover-table logic, which names transform's own watermarks table, moved into transform (leftovers.py).python -m tools.ci.standalone transformbuilds the wheel from the sdist and installs it outside the checkout with onlyorvex-contracts. It then runs transform's own tests, including the kit. Transform requires no interface contract, so no test adapter is needed. Its manifest lists norequires, and it runs withoutsemantic, the UI, BI or AI.Files and their purposes
packages/contracts/src/orvex/contracts/transform/__init__.py.../transform/interface.pyTransformprotocol),TransformErrorandTRANSFORM.../transform/models.py.../transform/steps.py$since.../transform/scheduling.py.../transform/analysis.py.../transform/columns.py.../transform/kit.py.../transform/kit_project.py.../transform/kit_models.py,kit_runs.py,kit_schedule.pypackages/contracts/src/orvex/contracts/config_file.pypackages/transform/src/orvex/transform/api.pypackages/transform/src/orvex/transform/leftovers.pypackages/transform/src/orvex/transform/module.tomlpackages/server/src/orvex/server/transform_calls.pypackages/server/src/orvex/server/command_caller.pyCallContextof a CLI command or scheduled run: user, folder, revisionpackages/server/src/orvex/server/project_files.pypackages/server/src/orvex/server/assembly.pyimplementation_ofpackages/server/tests/record_statements.pyrecord_statementsreplacedhistory,lake_open,leftovers,lineage,project,project_check,run,run_metadata,run_reads,run_steps,scheduler,validateorvex.transformimport left)auto_filter,graph,incremental*,lineage,model_tests,models,reconciliation,schedule)tools/checks/module_dependencies.pytzdatacounts as used wherezoneinfois imported and it is declaredtools/checks/data_readers.pycontracts, now that the configuration reader lives thereCase matrix
packages/transform/tests/test_api_kit.py::test_the_whole_kit_passes_in_process_and_serializedapi.Transformtest_api_kit.py::test_every_operation_is_implemented$since's time zone fails the kittest_api_kit.py::test_a_planted_wrong_implementation_fails_the_kit[_Floated-…]KIT_CASE_FAILED: since-value.lookback)test_api_kit.py::…[_Live-…]KIT_CASE_FAILED: lineage-project-key.slug)DECIMAL(38,4), floats never compared, NULL watermark not stored, DST inAmerica/New_York)record-statements.exact,measure-difference.exact,watermark-statements.exact,since-value.lookback,next-due.time-zone,schedule-plan.never-run-duepackages/server/tests/test_transform_calls.py::test_no_envelope_is_made_outside_a_commandtest_transform_calls.py::test_a_call_inside_a_command_carries_its_real_context,::test_one_command_keeps_one_context_per_projectpackages/server/tests/test_command_caller.py(4 cases)test_transform_calls.py::test_only_the_contract_operations_can_be_calledpackages/contracts/tests/test_transform_columns.py(3 cases)packages/server/tests/test_assembly.py::test_the_server_calls_the_one_installed_transform_implementation,::test_a_contract_no_installed_module_provides_is_refusedtests/tools/test_consumer_boundaries.py::test_no_consumer_uses_a_transform_internaltzdatawithoutzoneinfofail the standalone checktests/tools/test_standalone.py::test_tzdata_is_used_by_zoneinfo_only_when_declaredand the existing plantspython -m tools.ci.standalone transform(CI jobstandalone)How it was verified
verify fast: see the comment that follows (the machine is shared and loaded;fullruns in CI first, then once locally).pytest packages/server/tests packages/contracts/tests packages/transform/tests tests/tools/test_standalone.py tests/tools/test_consumer_boundaries.py. The six failures it found were all fixed at their cause: tests that patchedrun.record_statementsnow replace it at the contract. The re-run passes.mypy -p orvex.transform -p orvex.server -p orvex.contractsand the changed tests: clean.tools.contracts.interface_record --check: current.tools.ci.checks: matches the workflows.Size
Over the guide. The change is one coherent move, done per file and mechanically: types moved into the contract, call sites rewritten to the typed protocol, and the 12 server consumer files moved. The kit and the contract are new code. Review the contract (
interface.py,analysis.py) andapi.pyfirst, thentransform_calls.pyandproject_check.py.Time and CI-first errors
Filled in when CI finishes.
Not done here
serverstill names_meta.run_records_v2(run_reads._committed_snapshot), a transform table, as a string. The G2 storage-name check is not part of this step.request_contextis not used here.Laws touched
Law 11 (exact values): unchanged and proven by the kit (exact counts,
DECIMAL, floats refused). The fail-closed rule applies to every refused envelope.Contracts touched
transforminterface contract 1.0: new.orvex.contracts.config_file: moved from transform without change.Dependencies added
pyyaml6.0.3 becomes a dependency oforvex-contracts. It was already transform's, at the same pin and license, and it follows the configuration reader that moved.transformno longer declarespydanticorpyyaml, because it does not import them.Shared files changed
tools/ci/checks.toml(appendedstandalone-transform; thestandalonejob digest updated),.github/workflows/interfaces.yml(appended one step),tools/checks/data-reader-allowlist.toml(paths of the moved reader),tools/ci/test-durations.json(path of the moved test),packages/contracts/interface-record.json(regenerated).Risks
standalonejob digest will conflict with step 2 (store), which appends to the same job. Resolve it by recomputing the digest.Generated by Claude Code