Skip to content

Compress every prompt block, and guard the instruction against truncation - #115

Merged
tarekziade merged 2 commits into
mainfrom
tarek/instruction-budget-guard
Sep 7, 2026
Merged

tarekziade merged 2 commits into
mainfrom
tarek/instruction-budget-guard

Conversation

@tarekziade

@tarekziade tarekziade commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

Two things, both about how much of this reaches the model.

The instruction was being truncated

#114 pushed the output_mismatch instruction to 8,821 chars. serge's build_task_user_prompt head-truncates the instruction at MAX_INSTRUCTION_CHARS (8,000 at the time), so the 821 chars that fell off the end were the two bullets that PR existed to add — "the ACTUAL side of the assertion is off limits" and "a SHAPE or LENGTH mismatch is never expectation drift". Written, reviewed, merged, never delivered.

Every category's build_instruction output is now checked against that cap, with a failure message saying what truncation would cost. serge#118 raises the cap to 12,000 for headroom; this is the half that belongs here, where the blocks are written and where the next one will grow.

Compression

Only these strings are sent to the model — the # comments around them are not — so the rationale and dated precedents stay put and only the delivered text shrinks. 21,141 → 18,886 chars. Same rules, same thresholds, same PR numbers and examples: the 112 phrases the test module pins are all still there, and 328 passed.

block before after
_MISMATCH_GUIDANCE 6,178 5,031
_OOM_LOAD_GUIDANCE 4,624 4,333
_INSTRUCTION 2,641 2,192
_CULPRIT_GUIDANCE 2,110 2,041
_CRASH_GUIDANCE 2,058 1,969
_OOM_RETENTION_GUIDANCE 1,881 1,734
four small blocks 1,649 1,586

The largest single win is not prose. The trunk told the agent to go and read the root AGENTS.md / CLAUDE.md / .ai/AGENTS.md — which serge already injects verbatim into the system prompt as REPO CONVENTIONS. That paragraph paid for the conventions twice and bought three tool calls; it now says they are already in the prompt and points only at what the agent does need to open (the failing model's test file and its docs page).

Delivered instruction, worst case: output_mismatch 8,821 → 7,225 chars, inside even the old 8,000 cap.

Where the first load actually goes

For the worst-case task, with the compressed .ai/AGENTS.md:

system  9,876 + user 47,720 = 57,596 chars (~14.4k tokens)
  failure report (context)   40,000 chars — 69%
  conventions + instruction  10,363 chars — 18%

So the prompts are now the small half. If first-load size is still a concern, the lever is MAX_CONTEXT_CHARS (40,000) and how much traceback the triage packs into the report — not these blocks.

tarekziade and others added 2 commits September 7, 2026 09:53
#114 pushed the `output_mismatch` instruction to 8,821 chars. serge's
`build_task_user_prompt` head-truncates the instruction at
MAX_INSTRUCTION_CHARS (8,000 at the time), so the 821 chars that fell off the
end were the two bullets that PR existed to add — the guidance was written,
merged, and never reached the model.

serge raises the cap to 12,000; this is the half that belongs here, where the
blocks are written and where the next one will grow. Every category's
`build_instruction` output is checked against that number, with the failure
message saying what truncation would cost.

Sizes today: output_mismatch 8,821 · cluster 4,753 · crash 4,688 ·
load_error 3,217 · import_or_config 3,121 · trunk alone 2,641.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of these strings is sent to the model; the `#` comments around them
are not, so the rationale and the dated precedents stay where they are and only
the delivered text shrinks. Same rules, same thresholds, same PR numbers and
examples — the 112 phrases the test module pins are all still there.

  block                      before   after
  _INSTRUCTION                 2641    2192
  _MISMATCH_GUIDANCE           6178    5031
  _CRASH_GUIDANCE              2058    1969
  _CULPRIT_GUIDANCE            2110    2041
  _OOM_LOAD_GUIDANCE           4624    4333
  _OOM_RETENTION_GUIDANCE      1881    1734
  the four small blocks        1649    1586

The largest single win is not prose: the trunk told the agent to go and read the
root `AGENTS.md` / `CLAUDE.md` / `.ai/AGENTS.md`, which serge already injects
verbatim into the system prompt as REPO CONVENTIONS. That paragraph paid for the
conventions twice and bought three tool calls; it now says they are already in
the prompt and points only at what the agent does need to open — the failing
model's test file and its docs page.

Delivered instruction, worst case: output_mismatch 8,821 -> 7,225 chars, which
fits inside even the old 8,000-char cap with room to spare.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tarekziade tarekziade changed the title Guard the per-category instruction against serge's truncation cap Compress every prompt block, and guard the instruction against truncation Sep 7, 2026
tarekziade added a commit to huggingface/serge that referenced this pull request Sep 7, 2026
…118)

`build_task_user_prompt` head-truncates the instruction at
`MAX_INSTRUCTION_CHARS`, so an over-long one loses its **end** — and
transformers-ci appends its newest guidance last.

Its `output_mismatch` instruction (shared trunk + per-category block) is
**8,821 chars** after
[transformers-ci#114](huggingface/transformers-ci#114),
against a cap of **8,000**. The 821 chars that fell off were exactly the
bullets that change existed to add:

```
[... truncated, 821 chars omitted ...]
```
> cessor/tokenizer call) and check it against what the expectation
assumes: a flat list of N tokens, one waveform, one `.squeeze()` that
the test expects to collapse everything — all of those say batch size 1…

So the guidance was written, reviewed, merged, and never reached the
model. Nothing on either side noticed: the truncation is silent apart
from a marker in the prompt.

12,000 leaves room above today's largest block (sizes: output_mismatch
8,821 · cluster 4,753 · crash 4,688 · load_error 3,217 ·
import_or_config 3,121 · trunk alone 2,641). The other half is
[transformers-ci#115](huggingface/transformers-ci#115),
a test that fails when a category outgrows this number — that repo is
where the growth happens.

`24 passed` in `tests/test_prompts.py`.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tarekziade
tarekziade merged commit 3fdc7b4 into main Sep 7, 2026
@tarekziade
tarekziade deleted the tarek/instruction-budget-guard branch September 7, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant