Repository navigation
🧵 feat: Stream a Background Subagent Child Through Its Own Generation Job - #18169
Conversation
A generation job's idle clock advances only with emitted chunks, so a run that works without output for 20 minutes is reaped while it is still alive. touchJob refreshes the idle clock and the storage lifetimes an append would have extended, fenced on the generation epoch and limited to the replica that owns the runtime.
readSubagentContent folded public child frames into content parts inline. The fold is now a synchronous function a second caller can use, so a frame can be folded and published without yielding in between.
Job metadata gains subagentTaskId and subagentParentConversationId, set only by the subagent task store. A marked job is a child-thread job, so it stays out of the active chat list, and isBackgroundChildJob tells it apart from an event-bound child, whose job the chat controller owns.
createChildGenerationHost opens a generation job keyed by the child thread id and feeds it the child's public step frames. Text deltas that queue behind a slow write merge instead of dropping, the terminal state is claimed before the caller persists and the final event is published only afterwards, and no stream failure can fail the child: the run and its result carry on without a live view.
The subagent task store takes a childGenerations dependency. Once a child's thread and input row are durable it opens a job keyed by the thread id, feeds it each public frame, refreshes its idle clock on every lease renewal, and settles it around its own persistence, so a reader sees the final event only after the result row exists. A cancel, a failure and a job aborted from outside all end both the child and its stream. endpoints.agents.subagentActivity.generationJobs turns it off. A stream that cannot be opened or written never fails the child.
The host wiring builds the child generation host from the generation job manager and hands it to the subagent task store, so a background child's stream is reachable in the running server.
The chat abort route saves a user row and a partial response and upserts the conversation, and the steer route queues text only the chat controller drains. Both authorized on the job's owner alone, so either could act on a background child's job and write a chat-shaped history into a view-only thread, or accept a steer nothing would ever apply. Both now answer 409 for a job the subagent task store owns; the child is stopped and steered through its control endpoint.
The in-memory job store made room by deleting its oldest job whatever its status, so a full store could end a run somebody was still waiting on. It now takes the oldest finished job first. Also covers a job manager that shuts down while a background child runs: the child still persists its cancellation.
Against real Redis: a second replica resumes with everything the child has streamed, receives only newer deltas live, then the final event; a child that streams past the manager's 5,000-event early buffer before anyone attaches still resumes whole on both replicas; and the owner's touch keeps a silent child's job alive.
005df75 to
4e031c5
Compare
…18173) A child's result row carried its text and a bounded activity projection, not the content parts every agent message stores, so a reader had to rebuild the child's output from the projection. The row now also carries what the child streamed, on completion, failure and cancellation, in the standard shape. endpoints.agents.subagentActivity.generationContentBytes bounds it (4 MiB by default, 0 stores none). Content over the bound is left off, so the row always fits and keeps its text and projection as before.
|
@codex review the latest head, final review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54202272e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| export const subagentActivityConfigSchema = z.object({ | ||
| /** Run each background child as a resumable generation job keyed by its thread id, so its | ||
| * thread streams and resumes the way a chat does. Off leaves only the activity feed. */ | ||
| generationJobs: z.boolean().default(true), |
There was a problem hiding this comment.
Keep child generation jobs opt-in by default
For existing deployments that omit this new setting, .default(true) immediately enables an additional generation job for every background child, along with streamed-content persistence. Previously those deployments used only the activity feed. Root AGENTS.md requires new configuration levers to default to the existing behavior; default this toggle to false so operators can explicitly enable the new execution path.
AGENTS.md reference: AGENTS.md:L93-L95
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changing this one. The maintainer asked for this lever to ship on: the instruction for this work was that a change expected to be a net improvement defaults on, with the field as the off switch. generationJobs: false restores the previous behavior exactly, and the "runs without a job when the lever is off" and "stores text only when the stream is off" cases in subagentThreads.generation.spec.ts pin that. The cost while nothing reads the stream yet (one job write per streamed frame per background child) is stated in the PR description.
| const childRefusal = backgroundChildJobRefusal(job); | ||
| if (childRefusal) { | ||
| return sendGenerationJson( | ||
| res, | ||
| childRefusal.status, |
There was a problem hiding this comment.
Move the abort refusal branch into the TypeScript boundary
This adds a new validation branch and HTTP-response decision to the legacy CJS route. Root AGENTS.md explicitly says that /api holds wiring, not behavior, and applies that restriction to branches and validation added while editing existing CJS files. Although the refusal predicate lives in TypeScript, the new control flow remains here; move the refusal handling into the TypeScript request boundary and leave this file wiring the handler.
AGENTS.md reference: AGENTS.md:L80-L84
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 65173f3. The refusal is now createBackgroundChildAbortGuard in packages/api/src/agents/guard.ts: it resolves the abort target, reads the job, and answers 409 only for the owner's background child job. api/server/routes/agents/index.js registers it ahead of the abort handler and no longer contains the branch. Covered by the new background child abort guard cases in guard.spec.ts (owner, other user, other tenant, ownerless job, placeholder, missing id, store failure) and the existing route cases in abort.spec.js.
| ): Promise<void> { | ||
| this.settling = true; | ||
| await this.drainWithin(this.settleTimeoutMs); | ||
| const claim = this.detached ? null : await this.claim(aborted); |
There was a problem hiding this comment.
Terminalize the job even after detaching its stream
When a write fails, the backlog limit is exceeded, or settlement cannot drain within one second, detach() sets this.detached, so this branch permanently skips terminal claiming and cleanup even if the store subsequently recovers. Persistence still completes, but the job remains running until stale-job cleanup, subscribers receive no terminal event, and subsequent continuations cannot open their own streams because open() uses rejectActivePredecessor: true. The final cleanup call to settle() also skips the claim for the same reason. Detachment should stop frame publication without abandoning generation-fenced terminalization and resource cleanup.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f265873. settle now claims the terminal state whether or not the stream was detached, and publishes the full final event (the response is whole because frames keep folding after a detach). touch no longer skips a detached stream either, so its job is not reaped while the child still runs. The claim and the publish are bounded so a stalled store cannot hold the child's result. subagentGeneration.spec.ts: "keeps folding content after a write fails, and still persists" now asserts the job ends complete with a final event, plus "keeps a detached stream alive on touch until it is settled" and "does not hold the result behind a terminal claim that never answers".
| const live = job.status === 'running' || job.status === 'requires_action'; | ||
| if (!live && job.createdAt < oldestFinishedTime) { |
There was a problem hiding this comment.
Exclude pending terminal work from finished-job eviction
At capacity, this now preferentially evicts a newer terminal job even when it still has terminalPersistencePending or terminalHostActionPending, rather than selecting an older running job. A terminal status does not mean settlement is finished: child settlement claims complete before awaiting database persistence, and host-action failures retain terminal jobs for retry. Deleting such a record makes publishTerminalClaim() lose its finalization CAS and fall back to reconciliation, or permanently removes the record enumerated by getTerminalHostActionJobs() for retries. Include outstanding persistence, host-action, and schedule-reconciliation obligations when deciding whether a job is finished enough for preferential eviction.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 65173f3. Eviction now treats a terminal job as live while it has terminalPersistencePending, terminalHostActionPending, preserveForScheduleReconcile or providerDrained === false, so only a job with no outstanding obligation is preferred. jobEviction.spec.ts adds one case per obligation: the pending terminal job survives and the older running job is evicted, as before this PR.
| onJobAborted(): void { | ||
| if (!this.settling) this.input.onAbort(); |
There was a problem hiding this comment.
Distinguish stream write failures from child cancellation
With Redis enabled, a failed chunk append in GenerationJobManager.emitChunkNow() calls retireRuntimeAfterDurableFence(), which aborts the job controller before emitChunk() rejects. This handler immediately forwards that abort to the task store's cancellation callback, so the drain's intended catch-and-detach recovery runs only after the child has already been cancelled. A stream-store outage therefore stops model/tool execution instead of merely degrading the live view. Distinguish explicit deletion/shutdown cancellation from stream-runtime retirement, and cover an append failure through the real manager; the current flaky-manager test throws before entering this abort path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 65173f3, by removing the coupling rather than classifying aborts. The stream no longer forwards any abort of the job's controller to the task store: ChildGenerationInput.onAbort and the listener are gone. Deletion, shutdown and the control endpoint already stop a child through the task store's own cancellation, so an aborted, reaped or retired job now only ends the live view. childGeneration.stream_integration.spec.ts adds "leaves the child alone when the job store rejects a write, and still ends the job": the real manager on Redis with RedisJobStore.appendChunk rejecting, so the retirement path runs; content stays whole, persistence runs, and the job ends. subagentThreads.generation.spec.ts adds "lets the child finish and persist after its job is aborted from outside".
Four ways the stream could still reach the run it only observes. A frame that could not be copied threw into the child's own progress report. A job store that never answered held the child's start behind job creation and its result behind the terminal claim. And a stream that had stopped being written was never ended, so its job read as a live child for twenty minutes after the child finished. Frames are now copied defensively, opening and settling are bounded, a job that arrives after its child stopped waiting is ended, and a detached stream is still touched and still settled with its full final event.
Review round 1. A failed Redis append retires the job's runtime by aborting its controller, and the child stream forwarded every abort of that controller to the task store as a cancellation. A stream-store outage therefore stopped the child's model and tool work. The stream no longer signals the child at all: deletion, shutdown and the control endpoint already stop a child through the task store, and an aborted, reaped or retired job now only ends the live view. The in-memory store's capacity eviction preferred any terminal job, including one that still owed persistence, a host action, a schedule reconciliation or a provider drain. Those are no longer treated as finished. The chat abort route's refusal for a background child's job moves out of the CJS route into createBackgroundChildAbortGuard, registered ahead of it.
A background child's job outlives an owner that crashed, or a settlement that could not reach the store, for up to twenty minutes. A continuation of the thread in that window was refused its own stream, because the dead job still read as live. The caller of open holds the thread's lease, so a background child's job still marked live there has lost its owner: it is replaced and its readers are told to refetch. A live job that is not a background child's, such as an event-bound child paused for approval, is still never replaced.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65173f3d92
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } catch { | ||
| /** The route reads the same job and owns the answer for a store failure. */ | ||
| next(); | ||
| return; |
There was a problem hiding this comment.
Fail closed when the child-job guard lookup fails
If this lookup encounters a transient store error but the subsequent lookup succeeds, calling next() bypasses the background-child refusal entirely. The downstream /chat/abort handler in api/server/routes/agents/index.js independently reads the job and no longer checks backgroundChildJobRefusal, so it can abort the child's stream and execute its chat-shaped partial-message persistence while the child continues running. Forward the lookup error to the request error boundary, or ensure any retried lookup passes through the refusal check before abort processing.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in c7a21e4. A failed lookup now calls next(error), so the request ends at the error boundary and the route's chat-shaped abort never runs on a job the guard could not read. guard.spec.ts: "fails closed, without reaching the route, when the job lookup fails" asserts a 500 and that the route handler is not reached.
| const parts = lease.generation?.content().filter((part) => part != null); | ||
| if (parts == null || parts.length === 0 || limit === 0) return undefined; | ||
| try { | ||
| return Buffer.byteLength(JSON.stringify(parts), 'utf8') <= limit ? parts : undefined; |
There was a problem hiding this comment.
Budget streamed content against the complete message document
When generationContentBytes is configured to its permitted 8 MiB maximum, the independent content-size check can admit content that makes the complete message exceed MongoDB's 16 MiB document limit. For example, a 6 MiB streamed text response passes this check and the existing 12 MiB transcript limit, but persistResult() stores that text three times: in text, subagentTranscript.messagesJson, and the new content, producing a document over 18 MiB where the previous document was about 12 MiB. The save then fails and an otherwise successful child becomes a failed task. Account for the complete document's BSON size and omit the optional streamed content when it would exceed the remaining budget.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in c7a21e4. The streamed content is now budgeted against the row's other large fields: contentWithinDocument adds the content's serialized size to the text, subagentTranscript.messagesJson and the activity projection, and leaves the content off when the total passes 15 MiB, so the row keeps exactly what it stored before this PR. The final event carries the same decision. subagentThreads.generation.spec.ts: "leaves the content off when the whole row would outgrow the document" runs a 6 MiB child with generationContentBytes at its 8 MiB maximum; with the budget disabled that child ends as error, with it the task completes and the row has its text and no content.
…le Row Review round 2. The abort guard passed the request on when its job lookup failed, so a retried lookup in the route could run the chat-shaped abort on a child the guard never saw. A failed lookup now ends the request at the error boundary. A child's streamed content was checked against its own limit only. A result row also carries the text and the transcript, so content that passed the limit could push the document past MongoDB's 16 MiB and turn a successful child into a failed one. The content is now the optional field: it is left off whenever the row's large fields together would exceed the budget. Also accounts for the guard's job read in the two abort route specs that sequence it, and gives the limiter spec's api mock the guard.
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The task store settles a child's stream on its outcome path and again in its cleanup. The second call made another terminal claim that could only lose; the stream now remembers that it claimed and only persists. A frame that cannot be copied is still folded from the original, so the stored content stays whole when the stream detaches over it.
|
/quality risk scopes=api,server |
Quality requestOutdated: this request is for an earlier PR commit. Commit: Status: completed (success). Actions run (attempt 1).
Reports and scores:
Scores are advisory. Failed tests, tool errors and incomplete reports do not clear the gate. |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
The change-risk report showed the stream's fail-open branches untested: a terminal claim that lands after its wait, a claim or a final publish that throws, the settle timeout, the backlog limit, and a rejected touch. Each now has a case. The shared content fold gets direct cases for a closed step, a commentary phase and an applied steer.
|
/quality risk scopes=api |
Quality requestCommit: Status: completed (success). Actions run (attempt 1).
Reports and scores: Scores are advisory. Failed tests, tool errors and incomplete reports do not clear the gate. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
A background subagent child (
run_in_background) has a thread conversation but no generation job. Its live events reach a reader only through a bounded activity feed (the last 100 events or 64 KB, kept five minutes) and nothing durable is written until the run ends, so opening a child mid-run shows whatever tail the feed still holds. Switching chats does not have this problem because a chat has a generation job with a full chunk log and an anchored resume.This PR gives each background child a generation job keyed by its child thread id, the same convention chats and event-bound children already use.
GET /api/agents/chat/status/<threadId>andGET /api/agents/chat/stream/<threadId>?resume=truethen serve a child the way they serve a chat: a snapshot of everything so far, the gap, then live events, from any replica. When the run ends, the child's result row also stores what it streamed as standardcontent, so the row reads like any agent message.Nothing reads the new stream yet. The subagent panel still uses the activity feed, which keeps running unchanged beside the job; attaching the panel is the frontend work that follows. Until then the visible behavior is the same and each background child costs one extra job write per streamed frame.
Two levers on
endpoints.agents.subagentActivity:generationJobs(defaulttrue) turns the job off, andgenerationContentBytes(default 4 MiB serialized,0stores none) bounds the content stored on the row. The job defaults on at the maintainer's direction; off restores the previous behavior exactly.Builds on #18167 (merged), which keeps child-thread jobs out of
/chat/active.How it works
The task store does not import the job manager. It takes a
childGenerationsdependency, built in the server wiring fromGenerationJobManager.emitreturns immediately. While a write is in flight, consecutive text or reasoning deltas for one step are concatenated into one frame. The activity feed drops past 100 queued events.GenerationJobManager.touchJobrefreshes the clock and the storage lifetimes an append would have extended (JOB_TOUCH_LUAmirrors the append script's TTL block), fenced on the epoch and limited to the owning replica.openis called by the holder of the thread's lease, so a background child's job still marked live there has lost its owner (a crashed replica, or a settlement that could not reach the store). It is replaced and its readers are told to refetch. A live job that is not a background child's, such as an event-bound child paused for approval, is never replaced.POST /chat/abortandPOST /chat/steerauthorized on the job's owner alone. For a background child's job both now answer 409CHILD_THREAD_READ_ONLY: the abort route would save chat-shaped rows into a view-only thread, and a queued steer would never be drained. The abort answer iscreateBackgroundChildAbortGuardinpackages/api, registered ahead of the route; it reads the job once more than the route did, and a failed read ends the request instead of falling through.subagentTaskIdandsubagentParentConversationId, set only by the task store.generationContentBytesis left off, and so is content that would push the row's large fields (text, transcript, projection, content) past 15 MiB together, so the row always fits MongoDB's 16 MiB document and keeps what it stored before.Two supporting changes:
readSubagentContent's frame fold is extracted as a synchronouscreateSubagentContentFoldso a frame can be folded and published without yielding in between, and the in-memory job store evicts a job with no outstanding obligation before a live one at capacity.Choices worth a reviewer's eye:
completewithresponseMessage.error: true, so a reader gets a final event with the error row instead of a bare stream error.Type of change
Testing
Tested environments/configuration:
GenerationJobManagerClassAutomated tests:
subagentGeneration.spec.ts(20): job identity and marker, ordered delivery, late-subscriber content, merged deltas equal to the unmerged text, final only after persist, cancel, persist failure and nothing-persisted both publish a reconcile frame with no error text, a job aborted from outside leaves the stream folding and persisting, refused open, write failure keeps content whole and still ends the job, an uncopyable frame does not throw, a terminal claim that never answers does not hold the result, an open that takes too long is given up and the late job is ended, a background child's abandoned job is replaced with a refetch notice, and a live chat or event-bound job is never replaced.subagentThreads.generation.spec.ts(16): through the real task store and Mongo: mid-run resume, final after the result row is durable, cancel, failure without provider detail, a job aborted from outside leaves the child to finish, cancellation persists after the manager has shut down, continuation as a new generation of the same stream, lever off, stream unavailable, and the stored row'scontenton completion, cancel and failure, with text only when the stream is off, the content is over budget, or a 6 MiB child's row would outgrow the document.childGeneration.stream_integration.spec.ts(4, Redis): a second replica resumes with everything so far, receives only newer deltas, then the final event; a child that streams 5,202 separate chunks before anyone attaches (past the manager's 5,000-event early buffer, overflow marker asserted) resumes whole on both replicas; a rejectedappendChunkthrough the real manager leaves the child alone and the job still ends; touch keeps a silent job alive.guard.spec.ts: the abort guard for owner, other user, other tenant, ownerless job, placeholder, missing id, and a failed lookup that fails closed.jobTouch.spec.ts(5) and two Redis cases;jobEviction.spec.ts(6); steer and abort route cases; config defaults.subagentThreads.spec.ts(116),subagentContentunit and integration (70),steering/request.spec.ts(83),RedisJobStore.stream_integration.spec.ts(157),abort.spec.js(32).npx tsc --noEmitinpackages/api;npm run static-checks -- --against origin/dev.Measured on a local Redis, one machine, nobody attached:
emitChunkemit, 5,000 deltas in a tight loopemit, 300 deltas at about 100 per secondemit0.31 msemit, 2,000 deltas in bursts of 10 per 10 msAt an ordinary token rate nothing queues, so every delta is its own write; merging only helps when Redis falls behind. These are single-machine numbers, not a load test.
Not covered: a real provider run in the app, forking a child thread with the new rows, and anything in the browser, since no client reads the stream yet.