Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .github/klaud-candidate-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@ Treat candidate.source only as a published benchmark baseline. It is not authori
for current repository state, upstream releases or image compatibility. Resolve the exact
current family/image from the checkout, then independently inspect the actual bundled engine
source, official upstream releases and available images; stop if retired, ambiguous, updated
or owned. Normalize registry `/` versus enroot `#` spelling when comparing the same image.
or owned by another Klaud Cold PR. Open PRs on other branches never block a candidate.
Normalize registry `/` versus enroot `#` spelling when comparing the same image.
Use the canonical generator, inferencex-e2e/configs/runners.yaml and public OpenAPI/repository mappings
for all points, exact cluster routes and physical node demand; never invent aliases or
substitute sibling clusters. The planner already owns the family claim; leave claim refs
to the lifecycle helper. Recheck all open PRs before atomically creating the supplied
to the lifecycle helper. Recheck open Klaud Cold PRs (`klaud/auto-*`) before atomically creating the supplied
exact candidate branch at its base SHA. An existing branch means stop, not takeover,
unless KLAUD_RESUME=true.
Create one draft PR after a real change, titled `[Klaud Cold] English / 简体中文`.
Expand All @@ -40,7 +41,7 @@ Check changed defaults/semantics, renamed/removed options and hardware/CUDA/ROCm
Release notes alone are insufficient; retain source links and provenance uncertainty in
attempt comments. Pin mutable image tags by digest. Before GPU dispatch, inspect the
actual launcher/recipe's effective image, staged weights/config/tokenizer and mount paths.
Missing assets are readiness-blocked; never map to an older image or different weights.
Missing assets mean outcome missing-assets; never map to an older image or different weights.
When the model checkpoint itself is not staged on the target cluster, add
`reason-code: weights-missing` so the planner pauses that model and precision on that hardware.
Check the exact selected launch path for existing engine patches as well as proposed ones.
Expand All @@ -56,9 +57,9 @@ overlays, monkey patches or forked/rebuilt engine wheels, including pre-existing
If the shipped stack needs one, report incompatible. Preserve nodes:N, MTP chat templates,
workspace permissions and artifact contracts; follow the CODEOWNER checklist.

Before every PR/branch mutation or cancellation, re-read labels. `klaud-abort` means a
Before every PR/branch mutation or cancellation, re-read the PR. If someone else closed it, a
maintainer stopped this candidate: stop immediately without further changes; the job's
cleanup cancels your runs and closes the PR. Never add or remove that label yourself.
cleanup cancels your runs and deletes the branch. Never reopen it.

The planner checks capacity once, when it selects a candidate; never check or defer for
capacity inside this session. Klaud runs at the lowest scheduler priority: dispatch every
Expand Down Expand Up @@ -187,7 +188,7 @@ other outcome or any run except the verified final sweep.
Otherwise finish reports the failure/deferral, cancels owned work, confirms every job
terminal, removes sweep labels, drafts/closes the PR, deletes the candidate branch and releases
the family claim so a later wave may retry it. Pending cleanup means wait and retry finish.
Uncertainty is not incompatibility. Open PRs block the family until cleanup completes.
Uncertainty is not incompatibility. Your open PR blocks the family until cleanup completes.
Without an owned PR, report without a placeholder or deleting someone else's claim.

After finish succeeds, read verified $KLAUD_EVIDENCE/session-end.json and invoke
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/klaud-candidate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ jobs:
claude:
name: Klaud Cold
runs-on: ubuntu-latest
permissions:
contents: read
actions: read # Read workflow runs and artifacts.
pull-requests: read # Read PR metadata.
issues: write # Set display-only Klaud status labels on its PR.
env:
MAX_REPAIRS: '5'
KLAUD_BRANCH: klaud/auto-${{ inputs.candidate }}
Expand Down Expand Up @@ -154,6 +159,7 @@ jobs:
if: always() && steps.claude.outcome != 'skipped'
env:
GH_TOKEN: ${{ secrets.AGENT_PAT }}
KLAUD_STATUS_TOKEN: ${{ github.token }}
run: uv run --project "$INFERENCEX_TOOLING_PROJECT" --locked python -P -m infx.klaud recover-current
- name: Save sanitized candidate diagnostics
id: diagnostics
Expand Down
46 changes: 32 additions & 14 deletions .github/workflows/klaud-plan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,15 @@ name: Klaud Cold auto-sweep
run-name: >-
${{ inputs.resume-parent && format('Klaud Cold resume {0} {1} session {2}',
inputs.resume-parent, inputs.resume-candidate, inputs.resume-session)
|| github.event.schedule == '15,45 * * * *' && 'Klaud Cold wake'
|| (github.event.schedule == '15,45 * * * *' || inputs.wake) && 'Klaud Cold wake'
|| 'Klaud Cold auto-sweep' }}

# klaud-wake.yml dispatches a wake whenever a Klaud-owned run ends; one runs and one waits, so a
# burst of endings collapses into a single recovery pass. Every other run is its own group.
concurrency:
group: ${{ inputs.wake && 'klaud-wake' || github.run_id }}
Comment on lines +9 to +11

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Recovery passes can now overlap more often, which can push concurrent Klaud sessions past the MAX_CANDIDATES_PER_RUN cap of 5. Only dispatched wakes share the 'klaud-wake' concurrency group. Scheduled :15/:45 wakes and the 4-hourly planner each run in their own github.run_id group. Run-end wakes now arrive at any minute, so they can run recover alongside those passes. Fix: serialize every recovery pass, for example with job-level concurrency: {group: klaud-recover, cancel-in-progress: false} on the recover job. Schedule-only runs keep their own group today.

Why this was flagged

klaud-wake.yml:237 dispatches klaud-plan.yml -f wake=true whenever a Klaud-owned E2E run or klaud/auto-* Run Sweep completes, at any minute. klaud-plan.yml:11 puts only those runs in the 'klaud-wake' group. A scheduled 15,45 * * * * wake or the 0 */4 * * * planner uses github.run_id, so its recover job runs at the same time as a dispatched wake's recover. In lifecycle.py recover(), each pass counts running_sessions() (lifecycle.py:1285). Each then resumes approved sessions until running + resumed >= max_sessions (lifecycle.py:1293). Two overlapping passes both see, say, 4 running sessions, and each grants a different parked candidate, so 6 sessions run against a cap of 5. recovery_lease is per candidate (claims.py:126), so it does not stop this. On the base branch, recovery ran only at :00, :15 and :45, so passes rarely overlapped. The PR text says bursts collapse, but that holds only among dispatched wakes.

Verification: klaud-plan.yml:11 sets group: ${{ inputs.wake && 'klaud-wake' || github.run_id }}, so only dispatched wakes queue behind each other. In lifecycle.py recover(), each pass reads running = len(running_sessions(repository)) once (line 1285) and only counts its own resumed (line 1293). claims.recovery_lease (claims.py:126-155) is per candidate.

cancel-in-progress: false

on: # zizmor: ignore[concurrency-limits]
workflow_dispatch:
inputs:
Expand All @@ -20,8 +26,12 @@ on: # zizmor: ignore[concurrency-limits]
description: Number of the resumed session
type: string
default: ''
wake:
description: Recovery-only pass, dispatched when a Klaud-owned run ends
type: boolean
default: false
schedule:
- cron: '0 */3 * * *'
- cron: '0 */4 * * *'
- cron: '15,45 * * * *'

permissions:
Expand All @@ -41,6 +51,7 @@ jobs:
contents: read
actions: read # Read workflow runs and artifacts.
pull-requests: read # Read PR metadata.
issues: write # Set display-only Klaud status labels on its PRs.
outputs:
tooling-ref: ${{ steps.checkout.outputs.commit }}
blocked-families: ${{ steps.recover.outputs.blocked-families }}
Expand Down Expand Up @@ -81,24 +92,24 @@ jobs:
id: recover
env:
GH_TOKEN: ${{ secrets.AGENT_PAT }} # zizmor: ignore[secrets-outside-env]
KLAUD_STATUS_TOKEN: ${{ github.token }}
run: |
uv run --project "$INFERENCEX_PYTHON_PROJECT" --locked python -m infx.klaud recover --max-sessions "$MAX_CANDIDATES_PER_RUN"
echo "blocked-families=$(jq -c . "$RUNNER_TEMP/klaud-recovery.json")" >> "$GITHUB_OUTPUT"

review:
name: Review candidate overlap
needs: recover
if: github.event.schedule != '15,45 * * * *'
if: github.event.schedule != '15,45 * * * *' && !inputs.wake
runs-on: ubuntu-latest
permissions:
contents: read
actions: read # Read workflow runs and artifacts.
pull-requests: read # Read PR metadata.
issues: read # Read klaud-abort labels and Klaud outcome comments.
issues: read # Read who closed Klaud PRs and Klaud outcome comments.
env:
REVIEW_BATCH_SIZE: '64'
CANDIDATE_COOLDOWN_HOURS: '24'
STALE_PR_DAYS: '7'
outputs:
review: ${{ steps.review.outputs.structured_output }}
outcome: ${{ steps.review.outcome }}
Expand Down Expand Up @@ -135,7 +146,6 @@ jobs:
--directory "$RUNNER_TEMP/klaud"
--review-batch-size "$REVIEW_BATCH_SIZE"
--cooldown-hours "$CANDIDATE_COOLDOWN_HOURS"
--stale-pr-days "$STALE_PR_DAYS"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: klaud-review-input
Expand Down Expand Up @@ -180,9 +190,9 @@ jobs:
--json-schema '${{ steps.prepare.outputs.review_schema }}'
prompt: |
Use Read on candidates.json and open-prs.json under ${{ runner.temp }}/klaud/.
Review candidates in the supplied shuffled order against current recipes and the open-PR index,
including drafts and arbitrary branches. The index omits PRs idle for ${{ env.STALE_PR_DAYS }} days
(other than Klaud candidate PRs); those no longer own any family. Changed-file paths are already fetched;
Review candidates in the supplied shuffled order against current recipes and the open-PR index.
The index lists only open Klaud candidate PRs (klaud/auto-* branches). Other PRs never own a
family: ignore them, even when they edit the same image or recipe. Changed-file paths are already fetched;
inspect relevant PR bodies/diffs with direct gh pr view/diff commands. For API reads,
use gh api --method GET. Use Read/Glob/Grep for local inspection, not shell file commands
or Python. Run one allowed gh/git command per Bash call, without cd/env wrappers,
Expand All @@ -198,12 +208,10 @@ jobs:
a different family. For every proceed decision, resolve baseline-model to the exact display
model accepted by the public benchmark API using the supplied OpenAPI schema.
Target clusters and their capacity are resolved and checked in code, not by this review.
Mark duplicate when an open PR owns that family's image refresh or overlapping
Mark duplicate when an indexed Klaud PR owns that family's image refresh or overlapping
compatibility work, even with another image tag; unrelated work on the same hardware
is not a duplicate. A cross-cutting PR (launcher, telemetry, refactor or shared-schema
work spanning many families) is a duplicate only if its diff changes this family's
image or the exact engine flags/environment a refresh edits; touching the same recipe
file for unrelated settings is not overlap. Return the supplied schema: candidate-id, decision
is not a duplicate; touching the same recipe file for unrelated settings is not overlap.
Return the supplied schema: candidate-id, decision
(proceed/duplicate/uncertain), family (null if unresolved), pull-requests (overlapping
PR numbers), baseline-model (required for proceed; otherwise null), reason (brief evidence).
Missing evidence or ambiguity means uncertain.
Expand Down Expand Up @@ -315,6 +323,11 @@ jobs:
fail-fast: false
matrix:
candidate: ${{ fromJSON(needs.select.outputs.candidates) }}
permissions:
contents: read
actions: read # Read workflow runs and artifacts.
pull-requests: read # Read PR metadata.
issues: write # Set display-only Klaud status labels on its PR.
uses: $/.github/workflows/klaud-candidate.yml
with:
candidate: ${{ matrix.candidate }}
Expand All @@ -329,6 +342,11 @@ jobs:
if: >-
github.ref == 'refs/heads/main' && github.run_attempt == 1 && inputs.resume-parent != ''
&& github.actor == 'Klaud-Cold'
permissions:
contents: read
actions: read # Read workflow runs and artifacts.
pull-requests: read # Read PR metadata.
issues: write # Set display-only Klaud status labels on its PR.
uses: $/.github/workflows/klaud-candidate.yml
with:
candidate: ${{ inputs.resume-candidate }}
Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/klaud-wake.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Klaud Cold wake on run end
run-name: Klaud Cold wake on run end - ${{ github.event.workflow_run.display_title }}

# A parked Klaud session resumes only after recovery sees its owned runs end. Rather than wait
# for the next :15/:45 wake, start a recovery-only planner pass as soon as a Klaud-owned
# targeted run (e2e Test - klaud-*) or PR sweep (klaud/auto-* branch) completes. No checkout and
# no event data reaches a shell; the event only gates the job.
on: # zizmor: ignore[concurrency-limits]
workflow_run: # zizmor: ignore[dangerous-triggers] runs no code from the triggering run
workflows: [End-to-End Tests, Run Sweep]
types: [completed]

permissions: {}

jobs:
wake:
name: Wake Klaud recovery
if: >-
github.event.workflow_run.head_repository.full_name == github.repository
&& github.event.workflow_run.conclusion != 'skipped'
&& (startsWith(github.event.workflow_run.display_title, 'e2e Test - klaud-')
|| startsWith(github.event.workflow_run.head_branch, 'klaud/auto-'))
runs-on: ubuntu-latest
permissions:
actions: write # Dispatch the planner's recovery-only wake.
steps:
- name: Dispatch a recovery-only planner pass
env:
GH_TOKEN: ${{ github.token }}
run: gh workflow run klaud-plan.yml --repo "$GITHUB_REPOSITORY" --ref main -f wake=true
Loading
Loading