Repository navigation
📈 fix: Keep the Context Gauge Counting While a Reply Streams - #18487
Draft
danny-avila wants to merge 1 commit into
Draft
danny-avila wants to merge 1 commit into
danny-avila wants to merge 1 commit into
Conversation
Collaborator
Author
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The context gauge stands still for the whole of a reply and jumps to its final figure when the reply ends. Send a long reply and watch the gauge: it takes the count it had when the turn was sent, does not move while hundreds of tokens arrive, and lands on the total at the end.
The gauge only adds the output still arriving, and only uses the reading the server sends as a reply starts, while it considers the turn in progress: the response its usage is bound to has to be the one at the tail of the thread. A turn draws its reply under a placeholder id, and the first run step renames that row to the id the server stores it under. Usage stays bound to the placeholder, because every usage handler binds from the submission, which keeps the placeholder for the handlers that key on it. From the first run step on, no row has the bound id, the turn no longer counts as in progress, and the live figures are ignored until the final frame.
With this change the usage writers are told the id a run step gives the reply and follow the row to it. The reply being written stays the one in progress, so the gauge counts it as it arrives.
Measured on the mock e2e lane through one slow reply (gauge's used count at five points, then after it ended):
How it works
nameStepResponseis called wherever a run step is handed on: main chat's stream session, for a live step and for the steps replayed oncecreatedhas named the turn, and the event applier a coreChatruns, so a conversation driven that way keeps the same usage for the same frames. A step that reuses the preliminary id (USE_PRELIM_RESPONSE_MESSAGE_ID) or names none renames nothing.The submission is left as it is. The final handler, the regenerate merge and the abort paths find the turn's reply by the placeholder id it carries, so the id the row took is kept beside it, in the usage writers, rather than written over it.
bindResponsealready moved a turn's snapshot when the response id it was given changed; it now sees that change.Type of change
Testing
Reproduced in a real browser before the change, on a conversation just used and on one opened fresh: the gauge read the same number at chunks 5, 20, 60, 100 and 140 of a 160-chunk reply, then the settled total. After the change it rises at each point and ends on the same total.
The new scenario compares two readings taken while the reply is still running (the Stop control is on screen for both) and fails against a build without the change (
Expected: > 38,Received: 38). A first version of it waited for the higher reading, which let the end of the reply satisfy it; it passed without the fix, so I replaced it.The existing usage scenarios (
e2e/specs/mock/usage.spec.ts, seven) pass unchanged.Tested environments/configuration:
Automated tests:
packages/chat:npx jest(248 suites, 4008 tests),tsc --noEmit. New cases inreact/stream/usage/__tests__/handlers.spec.ts(usage follows the reply to the id it is named, through later frames; another turn's placeholder is not affected) and newcore/__tests__/nameStepResponse.spec.ts. The handler case fails with the alias removed.client:npx jest src/hooks/SSE src/hooks/Chat/__tests__/useTokenUsage.spec.tsx(12 suites, 496 tests).@scenario:context-gauge-follows-a-streaming-replyine2e/specs/mock/scenarios/gauge-follows-reply.spec.ts, run three times.