Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions prompts/cli/autopilot-core.prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -962,7 +962,14 @@ the evidence record.
or was killed by a signal, the wrapper's exit 127 (could not start), no output at
all, or a runner line in the log ("Shared test suite queue failed", "Test suite
execution timeout", "Test suite could not start", "Test suite ended by SIG…",
"Test suite descendants could not be confirmed", "Supply a suite command"). It
"Test suite descendants could not be confirmed", "Supply a suite command").
Read the shared queue ticket for the current suite checkout and attempt.
Report a live ticket waiting for admission as waiting, not executing tests.
A ticket interrupted before admission is a runner failure, never a test
verdict. Preserve its signal (fall back to the wrapper's signal if necessary)
and null test exit. Keep the most recent such interruption visible across
retries, separately from the replacement attempt. Wait for wrapper exit and
the usual ownership cleanup before replacing it. The existing budget applies: it
starts again in the same checkout, at most twice. On the 3rd failure, a runner that
died on all 3 tries (the deaths count over every try of this suite run) is a
`no-result` end; any other runner failure ends the job with a
Expand Down Expand Up @@ -1476,7 +1483,9 @@ runs until its end):
changes): the PR's Runtime lines and claims, the T01 result with the saved rows that
it read (Story, pass, current, time, tests, failed checks, refusal codes, sent; at
most 50) and the merge base read (commit, passing Stories, error), the replay and formal run with
each file's result, the full suite (state, exit, log name, failed tests), CI and review
each file's result, the full suite (state, exit, log name, failed tests and,
if known, `interruption: {ticket, signal, log}` for its most recent not-run
attempt), CI and review
states (with `builderFamily`, `reviewFamily` and `reviewIndependence` when known),
the PR's Evidence claim and the honest Evidence (above), and **one "Not
covered" list (B3)**: what the engine did not check at this head, for example each
Expand Down
15 changes: 12 additions & 3 deletions prompts/cli/test-suite-queue.prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ operator or worker command; it does not schedule tests or approve publication.
Export `suiteQueueRoot(env = process.env)`, returning the shared absolute queue
path selected below, or throwing for a relative path. Export
`suiteStatusWords(suite)`, returning one status line for the public status shape:
waiting with position, running, failure with reason, successful completion, or
waiting with position, running, interrupted before starting with reason, failure with reason, successful completion, or
queue unavailable with its cause.

Export native port adapter `executeSuite(command, {cwd, env, timeoutMs, onChild, onOutput, startGate = false, cpuIds, taskset}, ports = {})`,
Expand Down Expand Up @@ -194,7 +194,7 @@ atomically save the terminal receipt before returning or forwarding completion.

Receipts and `readSuiteProgress` expose the added fields above plus `exitCode`
(integer or null), `signal` (termination signal or null), `outcome` (`running`,
`completed`, or `unknown`), and `counts` (null without a complete TAP summary,
`completed`, `interrupted`, or `unknown`), and `counts` (null without a complete TAP summary,
else `{tests, suites, pass, fail, cancelled, skipped, todo}`, each a nonnegative
integer). Parse only a complete final top-level stdout TAP summary after a TAP 13 header
and a top-level plan. Later non-summary output or an unfinished final line
Expand All @@ -207,9 +207,18 @@ receipts and logs remain available through the queue. Default this port to true.
Older records without evidence fields remain readable and provide null values.

A later process observes the same ticket and recorder through the shared queue.
On native SIGINT or SIGTERM before admission, the waiting owner atomically
finishes its ticket with `outcome: interrupted`, `finishedAt` set to the current
time, the received `signal`, and a reason that says the tests did not run. Keep
`startedAt`, `recorderPid`, `childPid`, `exitCode` and `counts` null and leave
the raw output empty. Do not admit the interrupted attempt. The owner still
terminates by that signal, so its caller does not get a numeric test exit.
If the owner disappears without a terminal receipt, observation projects the
same not-run outcome for a dead, unstarted waiter without inventing a signal or
changing the stored record. Live or uncertain execution is not a not-run result.
A live recorder keeps custody even after its original caller dies. An identity
mismatch is not proof that the recorder still owns that PID. If neither recorder
nor suite group is live, report `outcome: unknown`, `exitCode: null` and unknown
nor suite group is live after admission, report `outcome: unknown`, `exitCode: null` and unknown
counts unless a terminal receipt exists. Do not invent an exit, infer failure
causes from output tails, repeat an uncertain command, or signal a recovered PID.
Live or uncertain descendants retain their slot. Status remains read-only.
Expand Down
54 changes: 40 additions & 14 deletions src/autopilot-core.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { githubFullLastPage } from './github-calls.mjs';
import { jobEnvironment, startAgentJob, readAgentJob, snapshotAgentJob, cleanupAgentJob, agentCommand, agentJobCpu, agentJobHangs, pddCheckFile, skillsCheckFile } from './agent-job.mjs';
import { fileReport, listReportDrafts, reportKind } from './autopilot-reports.mjs';
import { checkAutopilotBase } from './autopilot-base.mjs';
import { readSuiteProgress } from './test-suite-queue.mjs';
import { redactSecrets } from './secret-redaction.mjs';
import { runtimePreflightSlice } from './runtime-seam.mjs';
import { findOwners, headerEntries, isTestOwnedPath, resolveOwners } from './file-owner.mjs';
Expand Down Expand Up @@ -2798,47 +2799,61 @@ function queuedCommand(ctx, cwd, testCommand) {
? { command: '/bin/sh', args: ['-c', `${install} || { echo "Autopilot core: dependency install failed" >&2; exit 86; }; exec "$@"`, 'pdd-suite', process.execPath, ...suite] }
: { command: process.execPath, args: suite };
}
function suiteQueueProgress(fs, env, suite) {
if (!suite?.cwd || !Number.isFinite(suite.agent?.started)) return null;
const progress = readSuiteProgress({cwd:suite.cwd, env}, {fs});
return progress?.queuedAt >= suite.agent.started && progress.ticket !== suite.interruption?.ticket ? progress : null;
}
const interruptedSuiteWords = signal => `interrupted before starting${signal ? ` (${signal})` : ''}; tests did not run`;

async function suiteState(ctx, job) {
const start = (cwd, restarts = 0, deaths = 0, quiets = 0) => {
const start = (cwd, restarts = 0, deaths = 0, quiets = 0, interruption = null) => {
const suiteCommand = resolveSuiteCommand(ctx, job, cwd);
if (!suiteCommand) return { state: 'missing' };
// CI=true stops watch modes.
const command = queuedCommand(ctx, cwd, suiteCommand);
job.suite = { head: job.head, cwd, command: suiteCommand, restarts, deaths, quiets, agent: startJob(ctx, job, 'suite', cwd, '', command, { CI: 'true' }) }; ctx.saveJob(job); return { state: 'running' };
job.suite = { head: job.head, cwd, command: suiteCommand, restarts, deaths, quiets, ...(interruption ? { interruption } : {}), agent: startJob(ctx, job, 'suite', cwd, '', command, { CI: 'true' }) }; ctx.saveJob(job); return { state: 'running' };
};
const replace = async (cwdOf, restarts = 0, deaths = 0, quiets = 0) => {
if (job.suite?.agent && !await cleanupPart(ctx, job, job.suite.agent, 'suite')) return { state: 'running' };
const replace = async (cwdOf, restarts = 0, deaths = 0, quiets = 0, interruption = null) => {
if (job.suite?.agent && !await cleanupPart(ctx, job, job.suite.agent, 'suite')) return interruption ? {state:'interrupted', interruption} : { state: 'running' };
if (job.headTree?.commit === job.head && ctx.fs.existsSync?.(job.headTree.path) && !resolveSuiteCommand(ctx, job)) return { state: 'missing' };
return start(await cwdOf(), restarts, deaths, quiets);
const prior = job.suite?.head === job.head ? job.suite.interruption : null;
return start(await cwdOf(), restarts, deaths, quiets, interruption ?? prior);
};
if (!job.suite || job.suite.head !== job.head) return replace(() => checkout(ctx, job, 'suite'));
const suiteCommand = resolveSuiteCommand(ctx, job);
if (!suiteCommand) return { state: 'missing' };
const seen = ctx.jobs.read(job.suite.agent);
const queue = suiteQueueProgress(ctx.fs, ctx.ports.env, job.suite);
const waiting = queue?.state === 'waiting';
const interruption = queue?.outcome === 'interrupted'
? {ticket:queue.ticket, signal:queue.signal ?? seen.exit?.signal ?? null, log:basename(job.suite.agent.log)} : null;
if (seen.state === 'running') {
// The process rule: a suite that prints nothing new is a runner failure (retried); nothing new 3 times, or 3
// deaths in all, is a no-result, never a pdd fault.
const watch = await watchProcess(ctx, job, job.suite.agent, 'suite');
if (watch.state === 'ceiling') return { state: 'ceiling', cause: watch.cause };
if (watch.state !== 'quiet') return { state: 'running' };
if (interruption) return {state:'interrupted', interruption};
if (waiting || watch.state !== 'quiet') return { state: waiting ? 'waiting' : 'running' };
// Stop owners: a suite that hung before at this head, or hangs in a test file that the PR changes, is the issue.
const hang = await suiteHang(ctx, job);
if (hang) return { state: 'hung', hang };
const quiets = (job.suite.quiets ?? 0) + 1, restarts = job.suite.restarts ?? 0;
if (restarts < 2) return replace(() => job.suite.cwd ?? checkout(ctx, job, 'suite'), restarts + 1, job.suite.deaths ?? 0, quiets);
return { state: 'broken', dead: true, text: `${quietText(ctx, 'full test suite')} for head ${short(job.head)} (3 tries). Log: ${basename(job.suite.agent.log)} on this machine` };
}
if (seen.exit?.code === 0) { recordSuccess(ctx, job, job.suite.agent, 'suite', seen); return { state: 'pass' }; }
if (!interruption && seen.exit?.code === 0) { recordSuccess(ctx, job, job.suite.agent, 'suite', seen); return { state: 'pass' }; }
const log = readTail(ctx.fs, job.suite.agent.log);
const tail = cleanLogText(log.split('\n').slice(-15).join('\n'));
// A non-zero exit of the repository's test command is a red suite. Only a failure of the runner itself is not a
// test result: it could not start, it died or was killed, it timed out, or it wrote no output. Such a run starts
// again, at most twice, and then ends the job without spending a repair.
if (seen.state === 'dead' || seen.exit?.signal || [86, 127].includes(seen.exit?.code) || SUITE_RUNNER.test(stripAnsi(log)) || !log.trim()) {
if (interruption || seen.state === 'dead' || seen.exit?.signal || [86, 127].includes(seen.exit?.code) || SUITE_RUNNER.test(stripAnsi(log)) || !log.trim()) {
// "Died 3 times" counts every try of this suite run.
const deaths = (job.suite.deaths ?? 0) + (seen.state === 'dead' ? 1 : 0);
if ((job.suite.restarts ?? 0) < 2) return replace(() => job.suite.cwd ?? checkout(ctx, job, 'suite'), (job.suite.restarts ?? 0) + 1, deaths, job.suite.quiets ?? 0);
return { state: 'broken', dead: deaths + (job.suite.quiets ?? 0) >= 3, text: `the full test suite runner failed for head ${short(job.head)} (${seen.state === 'dead' ? 'the run died' : seen.exit?.signal ? `killed by ${seen.exit.signal}` : `exit ${seen.exit?.code}`}, 3 tries). Log: ${basename(job.suite.agent.log)} on this machine` };
if ((job.suite.restarts ?? 0) < 2) return replace(() => job.suite.cwd ?? checkout(ctx, job, 'suite'), (job.suite.restarts ?? 0) + 1, deaths, job.suite.quiets ?? 0, interruption);
const reason = interruption ? interruptedSuiteWords(interruption.signal) : seen.state === 'dead' ? 'the run died' : seen.exit?.signal ? `killed by ${seen.exit.signal}` : `exit ${seen.exit?.code}`;
return { state: 'broken', interruption, dead: deaths + (job.suite.quiets ?? 0) >= 3, text: `the full test suite runner failed for head ${short(job.head)} (${reason}, 3 tries). Log: ${basename(job.suite.agent.log)} on this machine` };
}
recordSuccess(ctx, job, job.suite.agent, 'suite', seen);
const clean = cleanLogText(log), failed = failingTests(clean);
Expand Down Expand Up @@ -3418,12 +3433,13 @@ function evidenceRecord(ctx, job, { lines = [], claims = [], ci = null, review =
const v = job.vectors, currentReview = job.review?.head === job.head ? job.review : null;
const revFamily = review?.reviewFamily ?? currentReview?.reviewFamily ?? null;
const revIndependence = review?.reviewIndependence ?? currentReview?.reviewIndependence ?? null;
const interruption = suite?.interruption ?? job.suite?.interruption;
return { schema: 'pdd.core-evidence.v1', issue: job.issue, pr: job.pr ?? null, head: job.head, agentHead: job.agentHead ?? job.head,
runtime: { lines: lines.map(l => ({ line: l.line, story: l.story, verdict: l.verdict })), claims, coverage: job.coverage ? { head: job.coverage.head, ok: job.coverage.ok === true, text: job.coverage.text ?? null, stories: job.coverage.stories ?? [], rows: job.coverage.rows ?? [], base: job.coverage.base ?? null, consumers: job.coverage.consumers ?? null } : null },
vectors: v ? { head: v.head, state: v.state, reason: v.reason ?? null, cut: v.cut ?? false, skipped: v.skipped ?? [],
control: v.control ? { code: v.control.code ?? [], fork: v.control.fork ?? null, error: v.control.error ?? null, state: controlResult(v)?.state ?? null } : null,
kinds: Object.fromEntries(RUN_KINDS.filter(kind => (v.files?.[kind] ?? []).length).map(kind => [kind, v.result?.[kind] ? { state: v.result[kind].state, tests: v.result[kind].tests, pass: v.result[kind].pass, fail: v.result[kind].fail, skipped: v.result[kind].skipped, files: (v.result[kind].files ?? []).map(f => ({ file: f.file, state: f.state, code: f.code, tests: f.tests, pass: f.pass, fail: f.fail, skipped: f.skipped })), failed: v.result[kind].failed ?? [] } : { state: v.state, files: v.files[kind].map(file => ({ file })) }])) } : null,
suite: suite ? { state: suite.state, head: job.suite?.head ?? null, exit: job.suite?.agent ? ctx.jobs.read(job.suite.agent).exit?.code ?? null : null, log: job.suite?.agent?.log ? basename(job.suite.agent.log) : null } : null,
suite: suite ? { state: suite.interruption ? 'interrupted' : suite.state, head: job.suite?.head ?? null, exit: !suite.interruption && job.suite?.agent ? ctx.jobs.read(job.suite.agent).exit?.code ?? null : null, log: job.suite?.agent?.log ? basename(job.suite.agent.log) : null, ...(interruption ? { interruption } : {}) } : null,
ci: ci ? { state: ci.state } : null,
review: review ? {
state: review.state,
Expand Down Expand Up @@ -4072,7 +4088,10 @@ async function mergeCheck(ctx, job) {
question: review.decision.question ?? `The review found a choice that issue #${job.issue} did not make: ${oneLine(item, 600)}. Which option do you want?` });
}
if (suite.state === 'missing') return end(ctx, job, cause('job', 'no-test-command', 'no full test command: set autopilot.testCommand in pdd.json'));
if (suite.state === 'broken') return end(ctx, job, suite.dead ? cause('job', 'no-result', suite.text, { owner: 'machine' }) : cause('pdd', 'suite-broken', suite.text));
if (suite.state === 'broken') {
if (suite.interruption || job.suite.interruption) saveEvidence(ctx, job, { lines, claims, ci, review, suite });
return end(ctx, job, suite.dead ? cause('job', 'no-result', suite.text, { owner: 'machine' }) : cause('pdd', 'suite-broken', suite.text));
}
if (review.state === 'fail') { reds.push(`Review FAIL from a fresh reviewer: ${(review.mustFix ?? []).join('; ') || 'see the review comment on the PR'}`); nonTestReds++; }
if (suite.state === 'fail') { reds.push(suite.text); if (suite.failure) reproduceChecks.push(suite.failure); }
} else if (job.review?.head === job.head && job.review.verdict === 'FAIL') {
Expand Down Expand Up @@ -4126,7 +4145,7 @@ async function mergeCheck(ctx, job) {
}
// Something still runs (CI that still changes, a reviewer or a suite process): wait, and the quiet clock restarts.
noteProgress(ctx, job, [job.head, ci.state, ci.sig, review.state, suite.state, vectors?.state, logSig(job.review?.agent && ctx.jobs.read(job.review.agent))].join('|'),
ci.state === 'pending' || review.state === 'running' || suite.state === 'running' || vectors?.state === 'running');
ci.state === 'pending' || review.state === 'running' || ['waiting', 'running', 'interrupted'].includes(suite.state) || vectors?.state === 'running');
if (!['green', 'none'].includes(ci.state) || review.state !== 'pass' || suite.state !== 'pass' || vectors?.state === 'running') { delete job.lastError; delete job.lastErrorTool; ctx.saveJob(job); return; }
// #3239 owner wait: files whose owners the path checks could not read on main (an unreadable pdd-bindings.json) keep
// the PR out of the lane. The wait is not progress, so a quiet period is a pdd step stop (one PDD report, no END).
Expand Down Expand Up @@ -6564,8 +6583,15 @@ export function coreStatus({ state, settings, sources = {}, guard = null }, port
if (mqLine) lines.push(mqLine);
const words = { now: (ports.now ?? Date.now)(), settings, longest: readLongest(fs, ports.root, (ports.now ?? Date.now)()) };
const running = (job, part, role) => part && !part.watch?.stopped && !part.recorded ? `; ${processWords(job, part, role, words)}${pddWords(fs, part)}` : '';
const suiteWords = (job) => {
if (!job.suite) return '-';
const queue = suiteQueueProgress(fs, ports.env, job.suite);
if (queue?.outcome === 'interrupted') return interruptedSuiteWords(queue.signal ?? readJson(fs, job.suite.agent?.exit)?.signal);
const current = queue?.state === 'waiting' ? `waiting for a check slot (position ${queue.position})` : `started${running(job, job.suite.agent, 'suite')}`;
return current + (job.suite.interruption ? `; last suite attempt ${interruptedSuiteWords(job.suite.interruption.signal)}` : '');
};
for (const job of jobs) {
if (job.phase === 'pr') lines.push(`#${job.issue} PR #${job.pr} at ${short(job.head)}: review ${job.review?.verdict ?? (job.review ? 'running' : '-')}${named(job.review?.reviewIndependence) ? ` (${job.review.reviewIndependence})` : ''}${job.review?.verdict ? '' : running(job, job.review?.agent, 'review')}, suite ${job.suite ? 'started' : '-'}${running(job, job.suite?.agent, 'suite')}, repairs ${job.repairs}${
if (job.phase === 'pr') lines.push(`#${job.issue} PR #${job.pr} at ${short(job.head)}: review ${job.review?.verdict ?? (job.review ? 'running' : '-')}${named(job.review?.reviewIndependence) ? ` (${job.review.reviewIndependence})` : ''}${job.review?.verdict ? '' : running(job, job.review?.agent, 'review')}, suite ${suiteWords(job)}, repairs ${job.repairs}${
job.lane?.state === 'final' ? '; merge lane: final step' : job.lane?.state === 'waiting' ? `; merge lane: waiting${job.lane.holder ? ` for PR #${job.lane.holder}` : ''} since ${new Date(job.lane.since).toISOString()}` : ''}${runtimeWords(job)}${poolWords(job)}${job.lastError ? `; last error: ${job.lastError}` : ''}`);
else if (job.phase === 'ending') lines.push(`#${job.issue} ending: ${job.end?.cause?.text}${job.lastError ? `; retry: ${job.lastError}` : ''}`);
else if (job.phase === 'released') lines.push(`#${job.issue} release pending: retry pdd autopilot release ${job.issue}`);
Expand Down
Loading
Loading