Skip to content

Audit successful runbook tests against their diagnostic contracts #189

Open

Description

@sushiHex

Problem and observed evidence

The #186 host session stopped before installation because the new preservation command emitted three GNU cp --no-clobber portability warnings. All copies and byte comparisons passed; the affirmative marker was printed. The procedure's unexpected-output rule nevertheless required a stop.

The credential-free test test_d2_preservation_copies_fixed_records_before_lr2_without_overwrite checks return code, stdout and retained bytes, but not accepting-path stderr. A green test therefore proved copy behavior, not the operator's complete diagnostic contract. Evidence: #186 (comment)

Bounded follow-up

Audit existing executable-runbook tests for this precise mismatch: successful command output that the operator's stop rule would reject but the test ignores. Start with the M8 host scripts and stock utilities on the declared Ubuntu runner. Distinguish expected diagnostics from exit status and affirmative domain markers; do not suppress stderr globally, accept arbitrary warnings, or weaken refusal rules.

Acceptance

  • Inventory the affected success-path assertions with source references, distinguishing observed gaps from speculation.
  • For selected active/successor procedures, assert the complete permitted stdout/stderr shape and demonstrate an assertion failure when an undeclared diagnostic is injected.
  • Exercise platform-sensitive utility behavior in the existing credential-free Linux lane where possible. Portable fake results must not be described as host observations.
  • Keep accepted runbook bytes frozen; corrections belong in a reviewed successor or explicit diagnostic disposition.
  • No VM, credential, vendor binary addition, model request or automatic resume framework.

The current #186 continuation addresses its own read-only checks and accepting-path diagnostics. This issue tracks the broader audit, not a prerequisite to that bounded continuation. Plans/ADRs and the normal review gate remain authoritative.

Agent: codex:constructicon-wave

Activity

  1. added
    needs-triageNeeds scope, reproduction, or authority review before work is accepted.
    on Oct 14, 2026
  2. sushiHex commented on Oct 14, 2026

    @sushiHex
    OwnerAuthor

    Additional reproduced instance while preparing the #186 continuation (#192): a local package compiler prepended HS H2 (which prints NO DRIFT) to C1's separate capture comparison, whose reviewed contract permits only its capture marker. Each extracted fragment was individually authentic; the composition was still wrong. Main-agent package review caught it before any upload or execution. No repository procedure change was needed.

    The rejected package set is retained and not approved for use. The corrected local compiler asserts each step's allowed source-block sequence and exact capture-body bytes. Reintroducing H2 in a disposable compiler copy produced the required AssertionError before creating the capture package; it was not a file-exists or harness failure. Mutation evidence digest: fef3c368ff1e7dceb9b1ba42f75cdd2d650cb764ab047843169eab5da9ee1d83.

    Include generated-package composition in this existing audit: authentic fragments and matching hashes do not establish the combined stdout/stderr contract. Prefer proving the actual composed step against its declared output; do not add a parallel runbook language or suppress diagnostics. This is follow-up evidence, not a claim that the broader audit is complete. No VM or vendor action occurred.

    Agent: codex:constructicon-wave

  3. sushiHex commented on Oct 16, 2026

    @sushiHex
    OwnerAuthor

    Progress from #210 (PR #211, merged as 4a84dfc), which covered a second instance of this class.

    • Inventory. M8-maintenance-lane-output.md records a source audit at C of every g11 boundary.
      • CONFLICT, five boundaries: S4 and the H7 active, S6b and S6c controls all print codex_lane's summary to stderr on success or on their expected refusal, and ER R1 adds its verdict on stdout.
      • CLEAN: most of the rest.
      • UNKNOWN: C10's run, whose only implicit stderr writers are asyncio's failure reports, plus m8-host-drift, sudo and the host Python.
    • Assertions. At those five boundaries the procedure now asserts the complete permitted shape: stderr byte-equal to the checked summary, the exit/fault rule, and an exact stdout verdict. Injected undeclared diagnostics fail, among them an extra line, a sudo warning and CRLF. Guard-removal tests are killed by assertion.
    • Prevention. The composition packager in feat(m8): assemble operator procedures from named fences, with lints #214 refuses any lane CLI call outside lane_capture, so this instance cannot be re-composed by hand.

    Still open here:

    • the P2 cp -n warning class on the declared Ubuntu runner;
    • the stock-utility audit for the remaining frozen runbooks;
    • a host measurement of the UNKNOWN rows.
  4. sushiHex commented on Oct 17, 2026

    @sushiHex
    OwnerAuthor

    Audit of the procedure tests against their diagnostic contracts, at 1043f48, prompted by q12's stop at C1 (a stubbed host line, NO DRIFT). Severity is relative to a live session.

    Blocks a live session

    • tests/test_m8_maintenance_q12_rehearsal.py:133 stubs m8-host-drift as print('NO DRIFT'). The stub and the declared expectation share that invented line, so they can never disagree. The fix is designed in the q13 issue.
    • tests/test_m8_maintenance_continuation.py:333-342 uses the same invented drift line, and its sudo stub prints nothing.
    • tests/test_m8_maintenance_session.py:194-209 (P2's cp --no-clobber) never asserts stderr == "" on success. That is this issue's original defect, still present in the frozen test. In fact the file has no empty-stderr assertion at all.

    Misleading

    • None of the N5 session tests asserts empty stderr on success: explicit refresh, g6, g7, g8, host, read, write and requalification. Their drift/sudo stubs print nothing or print marker words, so real host output is never modelled.
    • tests/test_m8_n5_g7_archive_custody.py:120-125 invents an AppArmor profile listing, labelled neither measured nor unmeasured.

    Tight already: test_m8_maintenance_g11_continuation.py, test_m8_lane_output.py, test_m8_maintenance_diagnosis.py.

    Host-only outputs the procedures assume:

    • m8-host-drift stdout, stderr and exit codes; D2:541 says the repository does not hold the tool;
    • that sudo -n true is silent;
    • that host Python 3.12 prints no warnings;
    • that cp --no-clobber is silent;
    • that ssh prints no banner. Only q12 sets LogLevel=ERROR; the older runbooks are frozen without it;
    • the format of the AppArmor profile listing.

    Proposed rule for successors:

    • A test may stub a host-only tool only with a label: measured, citing the private evidence, or synthetic. A synthetic stub must be paired with a case where a different plausible output makes the step fail.
    • Success paths assert stderr == "" wherever the contract says so.

    The frozen tests keep their bytes. Successors carry the fixes.

  5. sushiHex commented on Oct 18, 2026

    @sushiHex
    OwnerAuthor

    Baseline audit at f486bbe confirms q12 has 19 packaged steps but only combined H7/H8 and physical preflight run end-to-end through its transport. Other fragment and record tests are not composed accepting-path evidence. Source-only follow-up will add synthetic-labelled drift guard tests with plausible unexpected outputs, exact stderr assertions, and proof that refusal stops the next command. The broader utility/host audit remains open; no historical fixture is reclassified as measured. Refs #186 #226

  6. sushiHex commented on Oct 18, 2026

    @sushiHex
    OwnerAuthor

    Offline follow-up in progress on test/m8-isolated-qualification: an isolated, credential-free production CLI join for maintenance startup, checked emitted qualification evidence, descriptor publication/activation, and active startup. It will use real custody/store/launcher code with a synthetic native peer in the existing disposable Linux root lane. It is not a complete q13, C3/C4 or T5 rehearsal; no VM or vendor activity is authorized by this test. PR #235 remains separate and unchanged.

  7. 13 remaining items

  8. sushiHex commented on Oct 18, 2026

    @sushiHex
    OwnerAuthor

    Whole packaged C3/C4 source-proof follow-up merged in PR #241 as 879a71b; reviewed and merged trees match. Final evidence: #241 (comment) . Actual foundation artifact 12152474408 carries the accepting, historical-seal and wrong-floor cohorts, including two real 90-second holds and no application attempt. This closes that narrow proof gap, not private host readiness or live authority. Initial private q13 package now reproduces byte-identically with 16 steps and C9/application/C11 correctly deferred; focused package/readiness review is running. The small foundation runtime margin is separately tracked in #242; no timeout or proof weakened.

  9. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Starting the owner-approved bounded diagnostic-contract slice on test/189-success-diagnostics from c3323e1, session codex:constructicon-wave. Scope: kept-launch verification and the initial/qualify/active evidence-checker accepting paths, complete status/stdout/stderr assertions, and actual synthetic warning injection. Existing successor T5 is reused where small; no frozen runbook, runtime/source behavior or generic framework change. No VM, host query, credential, vendor/model execution or live authority. The prior whole-step seams remain established at their recorded boundaries, not reclassified as host measurements.

    Agent-Lane: codex:constructicon-wave

  10. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Bounded source-only diagnostic slice is committed locally as 5a388e7208b9720472c4167d26cf67cf48e6dd7d on test/189-success-diagnostics (base c3323e1), session codex:constructicon-wave. It is intentionally not pushed/opened yet: the approved 250/252 integration sequence lands first, then this branch will rebase once and receive exact-head CI.

    Selected omissions: LaunchHost.run discarded successful verify-launch stderr; initial/qualify/active evidence-checker acceptance asserted status/payload but omitted stderr. Existing success assertions now cover full permitted output, and actual synthetic child-warning controls bind otherwise-positive payload/status. The unchanged packaged kept-launch step runs through real T5; a warning refuses, is retained, and prevents an actual next shell command. No frozen runbook or production/runtime source changes. The g11 fragment helper already checked stderr and was not counted as another defect.

    Executed evidence: focused new/selected acceptance cohort 12 passed; helper/runtime/inventory cohort 216 passed, 91 Windows/Linux-only skips (140s); scope 60 passed; all 42 existing-plus-four-new procedure mutants assertion KILLED; touched Ruff and diff checks clean. These are portable synthetic proofs, not a private-host or vendor observation. Parent narrow source/test review found no further defect. Full exact-head CI and integration are pending, not claimed complete. The broader stock-utility/host audit remains open.

    No VM, credentials, model/vendor execution, resume or live authority. Ownership retained through the queued integration.

    Agent-Lane: codex:constructicon-wave

  11. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    The selected source-only diagnostic-contract slice merged in PR #253 as 606a22a from exact reviewed head 72d283f; merged and reviewed trees positively match (318cde77e6d78f5d38aa5b87e6fbe9ad4c40f300).

    Established: successful kept-launch capture and initial/qualify/active evidence-checker acceptance now assert the full permitted status/stdout/stderr contract. Real synthetic child warnings retain positive status/payload and fail those acceptance assertions. The actual packaged kept-launch step through unchanged T5 retains warning/status and prevents an actual following shell command. Four new guard mutants were assertion-KILLED; all 42 procedure inventory mutants passed their kill requirement. Existing g11 stderr coverage was not another omission and was not duplicated.

    Final exact-head verification, all mutation inventories, runner qualification, Windows discovery, and all four containment lanes passed. Narrow independent final-head review executed 12 selected tests and four new mutant controls, found no introduced findings, and verified manifest/prior-section preservation. The single connector review completed clean on that exact head. Evidence: #253 (comment)

    This issue deliberately remains OPEN: the broader stock-utility audit (including the original cp portability-warning class), older frozen-runbook limitations and private-host UNKNOWN outputs are not established by this slice. All new observations are credential-free/source or synthetic proofs, not current-host measurements. No VM, host query, credential, vendor/model run, live resume or new operational authority occurred. Frozen runbooks and production/runtime source stayed unchanged.

    Ownership of this bounded slice is complete; residual work stays in this issue rather than being claimed complete.

    Agent-Lane: codex:constructicon-wave

  12. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Starting the owner-approved bounded stock-utility diagnostic slice on test/189-stock-utility-diagnostics from 606a22a, session codex:constructicon-wave. The unchanged historical P2 preservation block will run over disposable files on the existing explicit Ubuntu 24.04 credential-free lane, retaining actual coreutils version and complete diagnostics in that lane artifact. Portable real-child controls will test the existing T5 success/refusal boundary and following-command stop, with an assertion-killed stderr-guard mutant. The historical test/runbooks stay unchanged. The current q13 archive already avoids this cp route and already has clean/warning tests, so it is not a new omission. Local Git Bash coreutils 8.32 is not Ubuntu evidence; the first exact-head CI establishes or refutes the runner-specific warning premise. Broader utility/host outputs remain outside this slice; no VM, vendor, credentials, retries, timeout increase or live authority.

  13. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Bounded stock-utility slice is now draft PR #256 at fd52751, integrated onto actual #255 merge 33b5e89. Final focused local cohort: 184 passed, one explicit Ubuntu-only skip; all 43 procedure mutants assertion-killed on byte-identical reviewed code, with independent narrow review clear. Historical P2/runbook/test bytes remain unchanged. The first required Ubuntu 24.04 probe and its retained actual version/diagnostic artifact are pending CI, not established by portable substitutions. Broader utility and private-host output obligations remain in this issue. No VM, vendor, credential, retry, timeout or live authority change.

  14. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Bounded stock-utility diagnostic slice completed in PR #256, squash-merged as 1d8c86ab30954b046598b6057c10402e5f6e56eb at 2026-10-19T20:14:29Z. The merged tree b263ec1760c73e0cbd083b31d3fa8e8e3d156b96 is positively identical to reviewed/tested head 5b1f431284a6486eef38b19c9502757243aec039; full-tree diff is empty and the required agent trailer is present once.

    Final-head CI: 7,214 passed, 450 skipped, plus successful mutation inventories and controller/structural proofs. All containment lanes passed. The required unprivileged Ubuntu 24.04 test ran rather than skipping. Foundation artifact 12217981562 was checked against its exact run/head metadata and entire intended JSON record: GNU coreutils 9.4; frozen producer digest; producer exit zero with exact marker and three complete pinned warnings; unchanged source and matching archive; T5 refusal and no next command. The portable warning guard is assertion-killed. Independent narrow review and exact-head connector review both found no introduced issue; all conversations/reviews/threads were read before the guarded merge.

    The first failed CI expectation and its exact absolute-prefix correction remain documented, not relabeled passing. Frozen runbooks and the historical test omission remain unchanged. This is a finite synthetic/disposable-CI observation and regression guard, not private-host, vendor, credential or general stock-utility qualification. The 450 skips are not positive proof. Broad #189 remains OPEN; no further implementation is implied.

  15. sushiHex commented on Oct 19, 2026

    @sushiHex
    OwnerAuthor

    Bounded triage at main 1d8c86ab30954b046598b6057c10402e5f6e56eb: defer further implementation until an authorized successor selects the remaining boundary.

    One pre-existing fragment-test omission was positively reproduced: the inherited C11 quiescence test checks status and the exact success marker, but its stderr assertion only excludes a sentinel. In a disposable, credential-free reproduction, that unchanged accepting test passed while its actual Bash child returned zero, printed the exact marker and emitted one explicitly synthetic warning on stderr. This is a test-evidence limitation, not an observed stock-utility or private-host diagnostic.

    The same otherwise-positive producer through unchanged T5's empty-stderr guard affirmatively refused; its capture retained producer status zero and the complete synthetic warning. No operational bypass was demonstrated. The q13 ledger already identifies C11's offline coverage as fragments rather than a composed rehearsal, and that historical session is complete/spent.

    #253 and #256 established their selected diagnostic slices; older frozen N5/q12 omissions remain historical limitations, while private-host unknowns require separately authorized measurements. If a future approved successor selects C11, carry this precise omission into whole-output/status assertions and an assertion-killed warning control. No new PR is proposed merely to change the historical fragment test. Only the focused reproduction above ran—not a full diagnostic audit. No shared source edits, VM, credentials, vendor/model operation or host observation occurred. Broad #189 remains open.

  16. sushiHex commented on Oct 23, 2026

    @sushiHex
    OwnerAuthor

    Concrete diagnostic-contract example from #40/#77: a V2 Claude verification failure produced only boot/drift markers and exit 1; the verifier report was remote and success-only readback did not fetch it. A separately prepared read-only diagnostic then stopped at its own G1 drift guard. That guard retains status/stdout/stderr remotely, but requiring the same guard to pass before negative readback blocks access to its failure evidence. Local reproduction of the exact emitted guard through T5 confirms correct acceptance/refusal envelopes; no transport defect or physical drift cause is established. Follow-up acceptance should distinguish operational authorization from fixed, bounded negative-evidence retrieval, preserving failure without granting retry/mutation authority. This is recorded backlog work, not permission to weaken any live execution gate.

  17. sushiHex commented on Oct 23, 2026

    @sushiHex
    OwnerAuthor

    Follow-up evidence from #40 comment6371475269: the historical host drift tool returned NO DRIFT while the sealed Claude launch runtime differed in four host-derived inputs (Git, libz, liblzma, CA bundle), making config and runtime metadata stale. Strict verification correctly refused but returned only installed:false, requiring a separate pinned read-only plan/inventory diagnostic. Keep security updates enabled and keep verification strict. A later scoped improvement should report bounded stage/reason/difference evidence from that same verifier law and explicitly treat changes to host-derived launch inputs as rebuild/requalification triggers. Do not add another runtime validator or silently accept old sealed copies. Deferred here while the existing reviewed refresh path restores Claude delivery.

  18. sushiHex commented on Oct 26, 2026

    @sushiHex
    OwnerAuthor

    Related bounded-diagnostic gap from the
    2026-10-26 stop:
    the remote preflight's conjunction checks operator HOME identity, canonical
    ancestry, ownership/permissions and fresh staging-path absence. It completed1
    with empty stdout/stderr, so the exact failed conjunct remains unknown.
    A copied namespace defect was separately verified in source; that is not a
    measurement identifying the guest's refusing conjunct.

    If this audit selects that surface later, prefer a bounded closed stage/reason
    projection only for positively observed failures, with explicit unknown for
    incomplete evidence. No raw path, environment, exception text or transport/native
    stream should be copied into public diagnostics. Preserve the refusal and
    no-resume rules; fake proofs should cover each check, ambiguous evidence and the
    whole public-field bound. This is follow-up evidence, not an expansion of current
    acceptance or an additional prerequisite for the narrow namespace correction.

  19. sushiHex commented on Oct 27, 2026

    @sushiHex
    OwnerAuthor

    Additional early-diagnostic gap identified while preparing the merged #311 operator tool: its qualification gate correctly hashes the actual local source bytes. A newline-translated Windows checkout can differ from the LF Git blobs installed on the controller, so that local gate refuses only after remote preparation. The byte mismatch was reproduced before live execution; it is not a vendor-authentication failure.

    Current preparation uses a positively verified Git-byte-exact source image and retains the strict qualification check. Follow-up within this audit: fail early before any remote phase when local controller-source bytes/module origins do not match the selected reviewed source image, and positively accept the matching image. Test both the matching and newline-translated cases through the isolated entry. Do not normalize evidence, weaken revision comparison, infer compatibility from matching module names, or add a general loader framework. No credentials, VM or vendor/model call is needed for this diagnostic.

    PR #311 separately delivered an actual credential-free generated accepting READ/withdrawal chain and a post-activation refusal/withdrawal chain. That closes those selected proof gaps, not this broader audit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageNeeds scope, reproduction, or authority review before work is accepted.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions