Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 23 additions & 13 deletions .github/klaud-candidate-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,15 @@ unless KLAUD_RESUME=true.
Create one draft PR after a real change, titled `[Klaud Cold] English / 简体中文`.
Start its body with only `<!-- klaud-baseline -->`; the report helper fills it.

When KLAUD_RESUME=true, an earlier session ended (ran out or was cancelled) before finishing
When KLAUD_RESUME=true, an earlier session parked, ran out or was cancelled before finishing
this candidate, and recovery started this session in a new run to continue it. The branch,
draft PR, frozen baseline (prepare-baseline returns it), attempt comments and runs are yours;
never recreate them. Check out origin/$KLAUD_BRANCH before editing. Recover the attempt
counter, repairs used and run IDs from your attempt comments; a commit pushed after the head
of your last attempt comment, other than the perf-changelog entry, counts as a used repair.
Inspect those runs (queued, running, preempted or finished) and continue. Repair budget,
infrastructure-retry limits and failure rules carry over.
Recovery starts a session only after every owned run ended, and it already reran the newest
run if someone preempted it. Inspect those runs (finished or preempted) and continue. Repair
budget, infrastructure-retry limits and failure rules carry over.

Compare the actual old/new bundled engine source at its tags/commits in vLLM, SGLang,
ATOM or TensorRT-LLM, including coupled dependencies and used flags/config parsers.
Expand Down Expand Up @@ -107,18 +108,25 @@ test-name=$KLAUD_TEST_NAME, and
Autosweeps cover only the family's AgentX (agentic-coding) points; never run fixed-seq-len points.
Smoke retains minimum-concurrency throughput and canonical representative eval concurrency;
it is startup evidence, not a full curve. Persist each run ID/head/attempt and typed progress
comment immediately, before waiting. Use `gh run watch --interval 60`, resume after tool
timeouts, and inspect jobs because queued workflows can contain active jobs and failed
benchmarks can leave evals running. Diagnose the FIRST server error, not teardown symptoms.
Empty aggregates or green collectors do not prove success.
comment immediately, before waiting. Inspect jobs because queued workflows can contain active
jobs and failed benchmarks can leave evals running. Diagnose the FIRST server error, not
teardown symptoms. Empty aggregates or green collectors do not prove success.

Never idle on queued work. Once every owned run you need is queued or running and nothing
else is left to do, publish an attempt report for each of them, then run `park` and end the
session as below. Recovery reruns the newest run if it is preempted and starts your next
session after every owned run ended, so parking costs no repair. You may watch a run that is
already running with `gh run watch --interval 60` for at most 30 minutes; park instead of
watching anything longer or still queued. When a hook says the session is near its time
limit, report progress and park, or finish if the outcome is already terminal.

One initial update plus $MAX_REPAIRS repairs TOTAL, including final-sweep repairs.
A confirmed transient client-download/runner/network retry does not consume a recipe
repair: keep the same head and classify it as infrastructure-retry, numbered 1 then 2 per attempt.
Do not repeat deterministic failures as infrastructure retries. Stop after validation,
exhausted repairs, or the same failure twice without progress.
Benchmarks may take three hours. Do not cancel healthy or queued work to fit the agent job
limit; leave it queued when the session ends, and a later session resumes it.
Benchmarks may take three hours. Never cancel healthy or queued work to fit the session limit;
park instead.

After smoke benchmarks AND selected evals pass, append one exact-family inferencex-e2e/perf-changelog.yaml
entry at the physical tail with this PR URL, preserving every prior byte. Set exactly
Expand Down Expand Up @@ -182,7 +190,9 @@ the family claim so a later wave may retry it. Pending cleanup means wait and re
Uncertainty is not incompatibility. Open PRs block the family until cleanup completes.
Without an owned PR, report without a placeholder or deleting someone else's claim.

After finish succeeds, read verified $KLAUD_EVIDENCE/outcome.json and invoke StructuredOutput
exactly once with that exact JSON object. Return no prose, Markdown, code fence, file path or
JSON-encoded string. Never forge completion markers or disable the Stop hook. Recovery can
preserve healthy children after SDK/job interruption; never abandon an unfinished session.
After finish succeeds, read verified $KLAUD_EVIDENCE/session-end.json and invoke
StructuredOutput exactly once with that exact JSON object. After park succeeds, do the same
with the `{"status": "parked"}` it writes there. Return no prose, Markdown, code fence, file
path or JSON-encoded string. Never forge completion or park markers or disable the hooks.
Recovery can preserve healthy children after SDK/job interruption; never abandon an
unfinished session.
15 changes: 11 additions & 4 deletions .github/workflows/klaud-candidate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ jobs:
cp -R "$RUNNER_TEMP/plan/$CANDIDATE" "$RUNNER_TEMP/klaud"
echo "KLAUD_EVIDENCE=$RUNNER_TEMP/klaud" >> "$GITHUB_ENV"
rm -r "$RUNNER_TEMP/plan"
outcome_schema=$(uv run --project "$INFERENCEX_TOOLING_PROJECT" --locked python -P -m infx.klaud outcome-schema)
echo "outcome-schema=$outcome_schema" >> "$GITHUB_OUTPUT"
session_end_schema=$(uv run --project "$INFERENCEX_TOOLING_PROJECT" --locked python -P -m infx.klaud session-end-schema)
echo "session-end-schema=$session_end_schema" >> "$GITHUB_OUTPUT"
- name: Check whether this session should run
id: gate
env:
Expand All @@ -110,6 +110,9 @@ jobs:
sleep 15
done
echo "path=$claude_cli" >> "$GITHUB_OUTPUT"
- name: Record when this session starts
if: steps.gate.outputs.run == 'true'
run: echo "KLAUD_SESSION_STARTED=$(date +%s)" >> "$GITHUB_ENV"
- name: Let Klaud Cold own the candidate PR
id: claude
if: steps.gate.outputs.run == 'true'
Expand All @@ -130,18 +133,22 @@ jobs:
"Stop": [{"hooks": [{
"type": "command",
"command": "cd \"$GITHUB_WORKSPACE\" && uv run --project \"$INFERENCEX_TOOLING_PROJECT\" --locked python -P -m infx.klaud check-stop"
}]}],
"PostToolUse": [{"hooks": [{
"type": "command",
"command": "cd \"$GITHUB_WORKSPACE\" && uv run --project \"$INFERENCEX_TOOLING_PROJECT\" --locked python -P -m infx.klaud check-deadline"
}]}]
}
}
claude_args: >-
--model 'claude-opus-5-5'
--max-turns 500
--json-schema '${{ steps.context.outputs.outcome-schema }}'
--json-schema '${{ steps.context.outputs.session-end-schema }}'
--add-dir '${{ runner.temp }}/klaud'
--allowedTools "Read,Write,Edit,Glob,Grep,Bash,WebFetch,WebSearch"
prompt: |
Read .github/klaud-candidate-prompt.md and follow it for the candidate in
${{ runner.temp }}/klaud/candidate.json. MAX_REPAIRS, KLAUD_BRANCH,
${{ runner.temp }}/klaud/candidate.json. MAX_REPAIRS, KLAUD_BRANCH, KLAUD_SESSION,
KLAUD_TEST_NAME, KLAUD_EVIDENCE and KLAUD_RESUME are provided in your environment.
- name: Reconcile agent-owned state
if: always() && steps.claude.outcome != 'skipped'
Expand Down
15 changes: 9 additions & 6 deletions .github/workflows/klaud-plan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Klaud Cold auto-sweep
run-name: >-
${{ inputs.resume-parent && format('Klaud Cold resume {0} {1} session {2}',
inputs.resume-parent, inputs.resume-candidate, inputs.resume-session)
|| github.event.schedule == '15,45 * * * *' && 'Klaud Cold wake'
|| 'Klaud Cold auto-sweep' }}

on: # zizmor: ignore[concurrency-limits]
Expand All @@ -20,7 +21,8 @@ on: # zizmor: ignore[concurrency-limits]
type: string
default: ''
schedule:
- cron: '0 */6 * * *'
- cron: '0 */3 * * *'
- cron: '15,45 * * * *'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After merging, Klaud can run many more concurrent agent sessions than the 5-per-wave cap, because resumes dispatched by wake runs are never counted against new candidates. Each wake pass resumes up to MAX_CANDIDATES_PER_RUN sessions. select subtracts only the resumed count from its own wave's recover job (slots = max(0, max_candidates - resumed), main.py:944). Fix: make new-candidate slots account for every live resumed session, for example by having select subtract the live Klaud sessions found by owned_activity, or by capping wake resumes to the headroom left by live sessions.

Why this was flagged

The new 15,45 * * * * cron (klaud-plan.yml:25) runs the recover job 48 times a day, and each run calls recover --max-resumes "$MAX_CANDIDATES_PER_RUN". That can dispatch up to 5 resume runs per pass (lifecycle.py:1166-1186). Wake runs skip review and select, so these resumes never reduce the next wave's slots. When the 6-hourly wave runs, sessions that were ready to resume have already been picked up by earlier wake passes, so its own resumed output is about 0. select then fills all 5 slots with new candidates (main.py:944) while the earlier wake-resumed sessions are still running. On the base branch, resumes happened only in the 6-hourly wave, and select's --resumed made resumed plus new sessions at most 5 per wave. After this change, the number of concurrent Claude sessions and GPU benchmark runs is limited only by how many candidates are open, and with 12 sessions and indefinite retention more of them stay open longer. No other code counts live sessions against the cap: owned_activity is used only by sweep_abandoned.

Verification: klaud-plan.yml:25 adds cron: '15,45 * * * *', on which select is skipped while recover still resumes up to 5. slots = max(0, max_candidates - resumed) at main.py:944 counts only the resumes made by the same run's recover job. Base comparison (f8eceff): recover ran only in the 6-hourly wave, so resumes and new candidates shared the same 5 slots.


permissions:
contents: read
Expand All @@ -42,7 +44,7 @@ jobs:
outputs:
tooling-ref: ${{ steps.checkout.outputs.commit }}
blocked-families: ${{ steps.recover.outputs.blocked-families }}
resumed: ${{ steps.recover.outputs.resumed }}
active-sessions: ${{ steps.recover.outputs.active-sessions }}
steps:
- id: checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -80,12 +82,13 @@ jobs:
env:
GH_TOKEN: ${{ secrets.AGENT_PAT }} # zizmor: ignore[secrets-outside-env]
run: |
uv run --project "$INFERENCEX_PYTHON_PROJECT" --locked python -m infx.klaud recover --max-resumes "$MAX_CANDIDATES_PER_RUN"
uv run --project "$INFERENCEX_PYTHON_PROJECT" --locked python -m infx.klaud recover --max-sessions "$MAX_CANDIDATES_PER_RUN"
echo "blocked-families=$(jq -c . "$RUNNER_TEMP/klaud-recovery.json")" >> "$GITHUB_OUTPUT"

review:
name: Review candidate overlap
needs: recover
if: github.event.schedule != '15,45 * * * *'
runs-on: ubuntu-latest
permissions:
contents: read
Expand Down Expand Up @@ -283,8 +286,8 @@ jobs:
KLAUD_DASHBOARD_API_KEY: ${{ secrets.DASH_API_KEY }} # zizmor: ignore[secrets-outside-env]
KLAUD_PR_REVIEW: ${{ needs.review.outputs.review }}
KLAUD_REVIEW_OUTCOME: ${{ needs.review.outputs.outcome }}
KLAUD_RESUMED: ${{ needs.recover.outputs.resumed }}
run: uv run --project "$INFERENCEX_PYTHON_PROJECT" --locked python -m infx.klaud select --max-candidates-per-run "$MAX_CANDIDATES_PER_RUN" --resumed "${KLAUD_RESUMED:-0}" --directory "$RUNNER_TEMP/klaud" --execution-file "$RUNNER_TEMP/review-execution.json"
KLAUD_ACTIVE_SESSIONS: ${{ needs.recover.outputs.active-sessions }}
run: uv run --project "$INFERENCEX_PYTHON_PROJECT" --locked python -m infx.klaud select --max-candidates-per-run "$MAX_CANDIDATES_PER_RUN" --active-sessions "${KLAUD_ACTIVE_SESSIONS:-0}" --directory "$RUNNER_TEMP/klaud" --execution-file "$RUNNER_TEMP/review-execution.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: klaud-ownership
Expand All @@ -302,7 +305,7 @@ jobs:
${{ runner.temp }}/klaud/review-diagnostics.json
${{ runner.temp }}/klaud/*/candidate.json
${{ runner.temp }}/klaud/*/baseline-preflight.json
retention-days: 14
retention-days: 90

candidates:
name: Run candidate ${{ matrix.candidate }}
Expand Down
Loading
Loading