ci: harden SDK test and publish verification (#33)

* ci: use cwd-safe stdlib Python tests

Run complete Python discovery from the python package directory in CI,
publish verification, and contributor instructions so imports resolve to the
checkout under test.

Remove the unused pytest development dependency and enforce the runner,
working directory, and documentation contract from the TypeScript policy
suite.

* test: make publish policy mutation CRLF-safe

Match both LF and CRLF when removing the npm publish environment in the
negative workflow-policy regression. This keeps the security mutation real on
Windows checkouts instead of silently testing the unchanged workflow.

* test: allow Python artifact failure check on slow runners

Give the clean-install failure regression 60 seconds to build a wheel and
exercise disposable Python environments. The Windows run completed near the
old 20-second ceiling, so the previous budget was flaky despite correct
behavior.

* docs: correct SDK 0.2.0 release evidence

Update the maintainer notes to the published npm and PyPI 0.2.0 state and
record that the release used the infrastructure-unavailable
require_live_canary=false override.

Keep the evidence boundary explicit: build, unit, packaging, security,
trusted-publish, and registry-install gates ran, but strict live-canary steps
were skipped and the release is not strict-live proof.

* fix: close SDK CI review gaps

Fail CI and publish on zero discovered Python tests even on Python 3.11, and
couple contributor cwd guidance to the exact unittest command in regression
coverage.

Align the live-canary guide with the documented infrastructure-unavailable
override while preserving the boundary that such a release is not strict-live
evidence.

* docs: preserve strict canary contract wording

Keep the established strict-child-canary and production-base-url contract
phrases intact while documenting the infrastructure-unavailable release
override.

* docs: record SDK CI hardening checkpoint

Record the verified branch, exact commits, full local gate totals,
Python-matrix caveat, and independent RELEASE verdict for handoff.

No push, publish, deployment, or production mutation was performed.

Author: jaberjaber23

.github/PULL_REQUEST_TEMPLATE.md

@@ -15,7 +15,7 @@
 
 - [ ] My code follows the style of this project (TS strict, no `any` in public surface; Python type hints + docstrings)
 - [ ] I have added tests for any new behavior
-- [ ] All tests pass locally (`pnpm test` for TS, `python -m pytest` for Python)
+- [ ] All tests pass locally (`pnpm --dir typescript test` for TS; from `python/`, `python -m unittest discover -s tests -v` for Python)
 - [ ] I have updated the relevant `CHANGELOG.md` files (typescript/ and/or python/)
 - [ ] I have NOT bumped the SDK version (maintainers handle that at release time)
 - [ ] I have NOT added any runtime dependencies (both SDKs are zero-dep by policy)

.github/workflows/ci.yml

@@ -143,7 +143,8 @@ jobs:
         working-directory: python
         run: |
           python -m pip install -e .
-          python -m pytest tests/ -v
+          python -c "import unittest; suite = unittest.defaultTestLoader.discover('tests'); count = suite.countTestCases(); print(f'Discovered {count} tests'); raise SystemExit(0 if count else 1)"
+          python -m unittest discover -s tests -v
 
   python-required:
     name: Python SDK

.github/workflows/publish.yml

@@ -114,9 +114,11 @@ jobs:
           node scripts/verify-npm-package.mjs "$TGZ"
 
       - name: Test Python
+        working-directory: python
         run: |
-          python -m pip install -e python
-          python -m pytest python/tests/ -v
+          python -m pip install -e .
+          python -c "import unittest; suite = unittest.defaultTestLoader.discover('tests'); count = suite.countTestCases(); print(f'Discovered {count} tests'); raise SystemExit(0 if count else 1)"
+          python -m unittest discover -s tests -v
 
       - name: Build Python artifacts
         run: python -m build python --outdir artifacts/python-local

AGENT-NOTES.md

@@ -8,11 +8,17 @@ history, no workspace identifiers, and no operational secrets.
 
 | Registry | Package | Source dir | Source version |
 |---|---|---|---|
-| npm | `@runinfra/sdk` | `typescript/` | `0.1.4` |
-| PyPI | `runinfra` | `python/` | `0.1.4` |
+| npm | `@runinfra/sdk` | `typescript/` | `0.2.0` |
+| PyPI | `runinfra` | `python/` | `0.2.0` |
 
-Registry latest remains `0.1.3` until the protected trusted-publish workflow
-publishes `0.1.4` from `main`.
+Registry latest is `0.2.0` on both npm and PyPI. Future releases must still use
+the protected trusted-publish workflow from `main`.
+
+The 0.2.0 release used the documented `require_live_canary=false` override
+because live-service canary infrastructure was unavailable. The workflow
+skipped strict readiness, artifact live-canary, and promotion-report steps;
+build, unit, packaging, security, trusted-publish, and registry-install gates
+still ran. Registry availability for 0.2.0 is therefore not strict live-canary evidence.
 
 ## What Ships
 
@@ -72,16 +78,16 @@ Important PyPI details:
 
 | Surface | Status |
 |---|---|
-| `client.chat.completions.create`, `client.responses.create` | Beta, contract-tested. Current 0.1.4 promotion artifacts are not strict-live green; publish requires fresh production artifact canaries with zero skipped or failed rows |
-| `client.embeddings.create` | Beta, contract-tested. Not strict live-canary verified in the current promotion artifacts |
+| `client.chat.completions.create`, `client.responses.create` | Beta, contract-tested. Published 0.2.0 artifacts are not strict-live verified because the infrastructure-unavailable override skipped those gates |
+| `client.embeddings.create` | Beta, contract-tested. Published 0.2.0 artifacts are not strict live-canary verified |
 | `client.images.generate` | Experimental, HTTP envelope matches the documented API but strict live canary coverage is still required |
 | `client.audio.speech.create` | Experimental, strict live canary coverage is still required |
 | `client.audio.transcriptions.create` | Experimental, strict live canary coverage is still required |
 | `client.voice.pipeline.create` | Experimental, pipeline-scoped route, not live-canary verified |
 | `client.webhooks.verifySignature`, `client.webhooks.constructEvent` | Local helpers, covered by unit tests |
 | Webhook delivery create/list | Not shipped and not exposed on the public SDK surface |
 
-Python remains sync-only in `0.1.4`. Do not claim async support until an
+Python remains sync-only in `0.2.0`. Do not claim async support until an
 `AsyncRunInfra` client has matching unit tests, streaming tests, live canaries,
 and clean-install coverage.
 
@@ -114,7 +120,10 @@ node scripts/verify-promotion-reports.mjs --readiness artifacts/sdk/live-canary-
 The publish workflow builds the npm tarball, Python wheel, and Python sdist
 once, uploads them as `runinfra-sdk-promoted-artifacts`, downloads the same
 artifacts for strict readiness and live promotion reports, recomputes hashes
-with `--artifacts-root .`, and the publish jobs publish only the downloaded `runinfra-sdk-promoted-artifacts` files. `dry_run=false` cannot bypass `promotion-gate`.
+with `--artifacts-root .`, and the publish jobs publish only the downloaded `runinfra-sdk-promoted-artifacts` files. `dry_run=false` cannot bypass `promotion-gate`. The infrastructure-unavailable
+`require_live_canary=false` override skips only the strict live-canary steps
+inside that gate and records that fact in the workflow summary; it does not
+turn the release into strict live-canary evidence.
 
 Clean artifact install/import now exercises the npm tarball, Python wheel, and
 Python sdist. The sdist path builds and imports in a disposable consumer

AGENTS-GOALs/2026-07-11-agent-sdk-ci-hardening-goal.md

@@ -0,0 +1,105 @@
+# SDK CI Test-Runner Hardening
+
+Date: 2026-07-11
+Agent: SDK hardening lane
+Repo: `runinfra-sdk`
+Worktree: `runinfra-sdk-ci-hardening-20260711`
+Branch: `fix/sdk-ci-hardening-20260711`
+
+## Goal
+
+Make every complete Python CI, publish, and contributor test command run the
+stdlib `unittest` suite from `python/`, so test imports resolve to the checkout
+under test instead of an installed package or sibling worktree. Preserve the
+existing package/runtime surface and trusted-publishing policy.
+
+Done means the Python suite, TypeScript suite, build, workflow/version/surface
+policy checks, and diff check pass from a fresh branch based on current
+`origin/main`; an independent hostile review finds no blocker; and the result
+is committed locally without pushing.
+
+## Initial Verified State
+
+- Fresh base: `b9e09353da858cdc5ec53a3600a1f345a89ee038` (`origin/main`).
+- The dirty detached SDK checkout and old PR #15 worktree are untouched.
+- `python -m unittest discover -s tests -v` from `python/` passed all 158 tests.
+- Two focused TypeScript policy regressions fail on the unmodified base:
+  stale pytest-version expectations and an LF-only publish-workflow mutation
+  that does not match Windows CRLF files.
+- npm and PyPI both report `0.2.0` as latest.
+- GitHub publish run `27575407723` published 0.2.0 with
+  `require_live_canary=false`; strict live-canary steps were skipped under the
+  documented infrastructure-unavailable override, while build, unit,
+  packaging, security, trusted-publish, and registry-install gates ran.
+
+## Scope
+
+- Existing CI and publish workflow Python test steps.
+- Python development requirements.
+- Pull-request and contributor test commands.
+- TypeScript policy regressions for Python tooling and Windows CRLF.
+- The slow Python artifact-failure test timeout, isolated as its own concern.
+- `AGENT-NOTES.md` release-state accuracy for 0.2.0.
+
+## Out Of Scope
+
+- SDK runtime behavior or public API changes.
+- Money, pricing, billing, or credit semantics.
+- Live-canary activation, infrastructure provisioning, publishing, pushing,
+  or merging.
+- Reusing or updating the old PR #15 branch.
+
+## Guardrails
+
+- Use the existing implementation paths, with no feature flag or parallel
+  runner.
+- Add no package dependency.
+- Stage explicit files only and commit with `git commit -F`.
+- Do not push or merge from this lane.
+
+## Checkpoint
+
+Implementation is complete on local branch
+`fix/sdk-ci-hardening-20260711`, based on
+`b9e09353da858cdc5ec53a3600a1f345a89ee038` (`origin/main`). The implementation
+HEAD is `ae4b5fcde541ac50f8895d42c8008cd6f4c0fa95`.
+
+Commits, oldest first:
+
+- `5f0fb6f0de49ab17cdcd1ba15bd252d3771f23b1` - cwd-safe stdlib Python runners.
+- `3c1823ef937283f85f5d8e2978e8fd9b2cf05c76` - Windows CRLF-safe policy mutation.
+- `06d0878a8727365c8c0b1346dc9aa4805bb2e12d` - isolated slow artifact-test timeout.
+- `cf6c150e80d95b64d19813769c74df0f4b9f41e8` - accurate 0.2.0 release evidence.
+- `f71b2c3d637d3c030fc7c48b1cdd43ee2cf05639` - close hostile-review gaps.
+- `ae4b5fcde541ac50f8895d42c8008cd6f4c0fa95` - preserve strict canary wording.
+
+Verified on the exact implementation HEAD:
+
+- Python 3.14.2 from `python/`: discovery guard found 158 tests; all 158 passed.
+- An empty-directory discovery probe exited 1, proving the guard rejects zero tests.
+- A fresh virtual environment installed the revised development requirements,
+  had no pytest package, and passed all 158 stdlib `unittest` tests.
+- TypeScript: all 258 Vitest tests passed, including workflow, cwd mutation,
+  Windows CRLF, release-evidence, and isolated-timeout regressions.
+- TypeScript no-emit typecheck and build passed.
+- Workflow policy, version sync, public-surface coverage, secret scan, Python
+  compileall, and `git diff --check` passed. Surface coverage reported zero
+  uncovered surfaces and zero uncovered rows.
+- The declared CI matrix remains Python 3.9 through 3.14. Only Python 3.14.2
+  was executable locally; the stdlib runner and guard are 3.9-compatible, but
+  Python 3.9 through 3.13 still require the remote CI matrix after integration.
+- A fresh fetch found `origin/main` unchanged at the base SHA. The branch was
+  six commits ahead and zero behind, so no rebase was required at that point.
+
+The first independent hostile review returned BLOCK with three findings:
+the CONTRIBUTING cwd mutation was not coupled to the command block,
+`LIVE-CANARIES.md` contradicted the documented override, and Python 3.11 raw
+discovery could pass with zero tests. All three were fixed and covered by
+regressions. A focused independent rereview of the exact committed range
+`b9e09353..ae4b5fc` then returned `RELEASE`: all three findings were closed and
+no remaining material issue was found. CodeRabbit was unavailable locally, so
+both hostile reviews used the configured read-only Codex reviewer.
+
+No push, merge, publish, registry mutation, live canary, deployment, or money
+change was performed. The old PR #15 worktree and dirty main checkout remain
+untouched.

CONTRIBUTING.md

@@ -19,7 +19,7 @@ pnpm build
 # Python
 cd ../python
 python -m pip install -e .
-python -m pytest tests/
+python -m unittest discover -s tests -v
 ```
 
 ## Branch + PR workflow
@@ -76,9 +76,11 @@ repo, so the PR title becomes the commit message — make it clean.
 
 Both SDKs have unit tests that mock the HTTP layer:
 - TypeScript: `typescript/src/index.test.ts` — runs via `vitest`.
-- Python: `python/tests/test_runinfra_sdk.py` — runs via `pytest`.
+- Python: `python/tests/test_runinfra_sdk.py` — runs via stdlib `unittest`.
 
 Add a test for any new behavior. Tests must not require network access.
+Run Python discovery from `python/`, as shown above, so imports resolve to the
+checkout under test instead of an installed package or sibling worktree.
 
 ## Releases
 

LIVE-CANARIES.md

@@ -109,10 +109,14 @@ satisfy strict promotion.
 In the trusted-publish workflow, `build-artifacts` creates the npm tarball,
 Python wheel, and Python sdist once and uploads them as
 `runinfra-sdk-promoted-artifacts`. `promotion-gate`, `publish-npm`, and
-`publish-pypi` download that same artifact bundle. A real publish cannot start
-the registry jobs until strict readiness/live reports pass for the downloaded
-artifacts and `verify-promotion-reports.mjs` confirms the same source digest
-and all-passed rows.
+`publish-pypi` download that same artifact bundle. By default, the registry
+jobs wait for strict readiness/live reports for those artifacts and for
+`verify-promotion-reports.mjs` to confirm the same source digest and all-passed
+rows. If live-service canary infrastructure is unavailable, a maintainer may
+explicitly dispatch with `require_live_canary=false`. The workflow records the
+override, skips only those strict live-canary steps, and still requires build,
+unit, packaging, security, trusted-publish, and registry-install gates. A
+release using the override is not strict live-canary evidence.
 
 ## Required Environment
 
@@ -194,9 +198,12 @@ canonical live canary matrix exactly,
 so a shortened self-consistent report cannot satisfy the gate. The report's
 candidate source file count must also match the canonical live-canary source
 file manifest.
-Promotion evidence must come from strict child canaries against `https://api.runinfra.ai/v1`;
-reports generated with any other custom `RUNINFRA_BASE_URL` are useful for
-staging smoke tests but cannot satisfy the real publish gate.
+Strict live-canary evidence must come from
+strict child canaries against `https://api.runinfra.ai/v1`; reports generated with any other custom
+`RUNINFRA_BASE_URL` are useful for staging smoke tests but cannot satisfy that
+evidence requirement. The infrastructure-unavailable override permits a
+registry release without this evidence, but it cannot be used to claim method
+or model live readiness.
 
 ## Matrix Rows
 

python/requirements-dev.txt

@@ -1,4 +1,3 @@
 build==1.4.0
-pytest==9.0.3
 typing_extensions==4.15.0
 twine==6.2.0

typescript/src/index.test.ts

@@ -2329,7 +2329,7 @@ class RunInfra:
       rmSync(tmp, { recursive: true, force: true });
       rmSync(tempRoot, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
     }
-  }, 20_000);
+  }, 60_000);
 
   it("keeps child canaries in parity for audio OpenAI parameter coverage", () => {
     const typescriptCanary = readFileSync(new URL("../../scripts/sdk-live-canary-typescript.mjs", import.meta.url), "utf8");
@@ -2915,6 +2915,13 @@ class RunInfra:
     expect(liveCanaries).toContain("RUNINFRA_ASR_FIXTURE_BASE64");
     expect(liveCanaries).toContain("RUNINFRA_VOICE_PIPELINE_AUDIO_BASE64");
     expect(agentNotes).toContain("`dry_run=false` cannot bypass `promotion-gate`");
+    expect(agentNotes).toContain("Registry latest is `0.2.0` on both npm and PyPI");
+    expect(agentNotes).toContain("documented `require_live_canary=false` override");
+    expect(agentNotes).toContain("Registry availability for 0.2.0 is therefore not strict live-canary evidence.");
+    expect(liveCanaries).toContain("explicitly dispatch with `require_live_canary=false`");
+    expect(liveCanaries).toContain("release using the override is not strict live-canary evidence");
+    expect(liveCanaries).not.toContain("A real publish cannot start");
+    expect(liveCanaries).not.toContain("cannot satisfy the real publish gate");
     expect(agentNotes).toContain("Clean artifact install/import now exercises the npm tarball, Python wheel, and");
     expect(agentNotes).toContain("node scripts/verify-github-security-status.mjs --repo RightNow-AI/runinfra-sdk");
     expect(agentNotes).toContain("the publish jobs publish only the downloaded `runinfra-sdk-promoted-artifacts` files");
@@ -3252,13 +3259,51 @@ class RunInfra:
   it("keeps Python test tooling compatible with the declared Python floor", () => {
     const pyproject = readFileSync(new URL("../../python/pyproject.toml", import.meta.url), "utf8");
     const requirements = readFileSync(new URL("../../python/requirements-dev.txt", import.meta.url), "utf8");
+    const publish = readFileSync(new URL("../../.github/workflows/publish.yml", import.meta.url), "utf8");
+    const ci = readFileSync(new URL("../../.github/workflows/ci.yml", import.meta.url), "utf8");
+    const pullRequestTemplate = readFileSync(new URL("../../.github/PULL_REQUEST_TEMPLATE.md", import.meta.url), "utf8");
+    const contributing = readFileSync(new URL("../../CONTRIBUTING.md", import.meta.url), "utf8");
+    const unittestCommand = "python -m unittest discover -s tests -v";
+    const nonzeroDiscoveryGuard = "python -c \"import unittest; suite = unittest.defaultTestLoader.discover('tests'); count = suite.countTestCases(); print(f'Discovered {count} tests'); raise SystemExit(0 if count else 1)\"";
+    const normalizedCi = ci.replace(/\r\n/gu, "\n");
+    const normalizedPublish = publish.replace(/\r\n/gu, "\n");
+    const normalizedContributing = contributing.replace(/\r\n/gu, "\n");
 
     expect(pyproject).toContain('requires-python = ">=3.9"');
     expect(pyproject).toContain('requires = ["setuptools==82.0.1"]');
     expect(pyproject).not.toContain("setuptools>=");
-    expect(requirements).toContain("pytest==8.4.2");
     expect(requirements).toContain("typing_extensions==4.15.0");
-    expect(requirements).not.toMatch(/^pytest==9\./mu);
+    expect(requirements).not.toMatch(/^pytest(?:[<=>~!]|$)/mu);
+    expect(normalizedCi).toContain([
+      "      - name: Test",
+      "        working-directory: python",
+      "        run: |",
+      "          python -m pip install -e .",
+      `          ${nonzeroDiscoveryGuard}`,
+      `          ${unittestCommand}`,
+    ].join("\n"));
+    expect(normalizedPublish).toContain([
+      "      - name: Test Python",
+      "        working-directory: python",
+      "        run: |",
+      "          python -m pip install -e .",
+      `          ${nonzeroDiscoveryGuard}`,
+      `          ${unittestCommand}`,
+    ].join("\n"));
+    expect(`${publish}\n${ci}`).not.toContain("python -m pytest");
+    expect(`${publish}\n${ci}`).not.toContain("-s python/tests");
+    expect(pullRequestTemplate).toContain(`from \`python/\`, \`${unittestCommand}\``);
+    const contributingPythonBlock = [
+      "# Python",
+      "cd ../python",
+      "python -m pip install -e .",
+      unittestCommand,
+    ].join("\n");
+    expect(normalizedContributing).toContain(contributingPythonBlock);
+    const wrongCwdContributing = normalizedContributing.replace("cd ../python", "cd ..");
+    expect(wrongCwdContributing).not.toBe(normalizedContributing);
+    expect(wrongCwdContributing).not.toContain(contributingPythonBlock);
+    expect(contributing).toContain("Run Python discovery from `python/`");
   });
 
   it("installs Python build tooling before publish workflow TypeScript tests", async () => {
@@ -3310,7 +3355,10 @@ class RunInfra:
     expect(evaluateWorkflowPolicy({ publish: realPublishWithoutDryRunGuard, ci, hasCustomCodeqlWorkflow: false }).find((check) => check.label === label)?.ok)
       .toBe(false);
 
-    const realPublishWithoutEnvironment = publish.replace("    environment: npm\n", "");
+    const realPublishWithoutEnvironment = publish.replace(
+      /    environment: npm\r?\n/u,
+      "",
+    );
     expect(realPublishWithoutEnvironment).not.toBe(publish);
     expect(evaluateWorkflowPolicy({ publish: realPublishWithoutEnvironment, ci, hasCustomCodeqlWorkflow: false }).find((check) => check.label === label)?.ok)
       .toBe(false);