Skip to content
Draft
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions client/src/Providers/SideChatHostContext.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,19 @@ import { useCallback, useMemo } from 'react';
import { useStore } from 'jotai';
import { useQueryClient } from '@tanstack/react-query';
import { createGenerationTransport } from '@librechat/chat';
import { filesFamily, useFollowChat, SIDE_CHAT_PANE } from '@librechat/chat/react';
import {
dataService,
QueryKeys,
isAgentsEndpoint,
isEphemeralAgentId,
} from 'librechat-data-provider';
import {
filesFamily,
useFollowChat,
SIDE_CHAT_PANE,
useUsageHandler,
toResumableUsage,
} from '@librechat/chat/react';
import type { Agent, TMessage, TConversation, ChatFinalFrame } from 'librechat-data-provider';
import type { FollowedChat, SideChatHost, FollowChatOptions } from '@librechat/chat/react';
import { isTemporaryConversation, proveFetchedReplyByIdentity } from '~/utils';
Expand Down Expand Up @@ -82,6 +88,13 @@ function useChat(
* the app's file, memory and preview caches.
*/
const onAttachment = useAttachmentHandler(queryClient);
/**
* The conversation's token usage, kept by main chat's own writers: the context reading the
* server sends as a reply starts, the output arriving meanwhile, and the totals at its end.
* It is what the composer's gauge reads, by conversation.
*/
const usageHandlers = useUsageHandler();
const usage = useMemo(() => toResumableUsage(usageHandlers), [usageHandlers]);
/**
* What lets a reply read here be acknowledged, as main chat's is: the reply a turn just ended
* on is stamped on the conversation wherever it is listed, paired with the list that holds
Expand All @@ -90,6 +103,8 @@ function useChat(
const onSettled = useCallback(
(final?: ChatFinalFrame) => {
if (final == null) {
/** No reply is on its way in what was just read, so no estimate of one is kept. */
usageHandlers.resetLive({ conversation: { conversationId: id } });
Comment on lines 105 to +107

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear side-chat usage on terminal paths without a final frame

When a turn sent from the side composer receives a terminal error frame after streaming output or token usage, this reset is never reached: createEventApplier only flushes message writes for errors, and useFollowChat's conclude() does not call onSettled for an owned turn without a final frame. The newly connected writers therefore leave the active response, live counters, and pending usage populated. useTokenUsage continues adding that pending usage to totals, and the next turn inherits it because neither bindResponse nor contextHandler clears pending usage. Wire terminal failure cleanup separately from history settlement, with a regression test covering an error after usage has arrived.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, fixed in 8c883b3.

A turn sent from the side composer that ends with no final frame (an error frame, or a stop before the server kept anything of it) now drops what was counted for its reply: useFollowChat's conclude() calls usage.resetLive on that path, which toResumableUsage maps to the same resetLive main chat's session runs on its error frame (core/resumable/failure.ts). A turn that ends on its final frame is still settled by finalizeUsage alone.

Test: driven.spec.tsx › "drops what was counted for a reply that fails, and keeps it for one that ends" runs the real usage writers against a store, sends a turn, folds a token_usage frame, then an error frame, and reads the pending usage and the active response back. Without the call it fails with the 40 output tokens and one folded event still pending; with it, both are cleared, and a following turn that ends on its final frame never triggers the reset.

const messages = queryClient.getQueryData<TMessage[]>([QueryKeys.messages, id]);
if (messages != null) {
proveFetchedReplyByIdentity(queryClient, id, messages);
Expand All @@ -106,13 +121,13 @@ function useChat(
updatedAt: settled.updatedAt,
});
},
[queryClient, id],
[queryClient, id, usageHandlers],
);
/** A turn here is said to a screen reader as main chat's is: started, still going, done. */
const { announcePolite: announce } = useLiveAnnouncer();
const beside = useMemo(
() => ({ onAttachment, onSettled, announce }),
[onAttachment, onSettled, announce],
() => ({ onAttachment, onSettled, announce, usage }),
[onAttachment, onSettled, announce, usage],
);
return useFollowChat({
id,
Expand Down
45 changes: 45 additions & 0 deletions client/src/Providers/__tests__/SideChatHostContext.spec.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import React from 'react';
import { RecoilRoot } from 'recoil';
import { getDefaultStore } from 'jotai';
import { contextSnapshotFamily } from '@librechat/chat/react';
import { act, renderHook, waitFor } from '@testing-library/react';
import { QueryKeys, EModelEndpoint } from 'librechat-data-provider';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
Expand Down Expand Up @@ -364,6 +366,49 @@ describe("the app's side chat host", () => {
expect(announcePolite).toHaveBeenCalledWith({ message: 'end', isStatus: true });
});

it("keeps the conversation's usage as main chat keeps it, for the composer's gauge", async () => {
mockStreamStatus.mockResolvedValue({
active: true,
streamId: 'side-conversation',
createdAt: 2000,
generationProtocolVersion: 2,
});
mockMessages.mockResolvedValue([asked]);
const { attachments } = setup();
await waitFor(() => expect(attachments).toHaveLength(1));
const emit = (event: ChatEvent) => act(() => attachments[0].options.onEvent(event));
const usage = getDefaultStore();
expect(usage.get(contextSnapshotFamily('side-conversation'))).toBeNull();

emit({ type: 'open' });
emit({
type: 'created',
data: { created: true, message: asked, turn: { responseMessageId: 'response-1' } },
} as ChatEvent);
/** What the server reads the context at as the reply starts. */
emit({
type: 'context_usage',
data: {
breakdown: {
maxContextTokens: 8000,
instructionTokens: 120,
systemMessageTokens: 100,
dynamicInstructionTokens: 0,
toolSchemaTokens: 20,
summaryTokens: 0,
toolCount: 1,
messageCount: 1,
messageTokens: 340,
availableForMessages: 7880,
},
},
} as ChatEvent);

await waitFor(() =>
expect(usage.get(contextSnapshotFamily('side-conversation'))).not.toBeNull(),
);
});

it('stops a generation through the app, by its conversation and the stream it names', async () => {
mockStreamStatus.mockResolvedValue({
active: true,
Expand Down
46 changes: 10 additions & 36 deletions client/src/hooks/SSE/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,17 @@ import { useMemo, useState, useCallback } from 'react';
import { useSetAtom } from 'jotai';
import { useQueryClient } from '@tanstack/react-query';
import { getStreamStartFailureText } from '@librechat/chat';
import { useUsageHandler, useEventHandlers, showStopButtonByIndex } from '@librechat/chat/react';
import {
useUsageHandler,
toResumableUsage,
useEventHandlers,
showStopButtonByIndex,
} from '@librechat/chat/react';
import type { ResumableRun, ResumableEvents, ResumableUsage } from '@librechat/chat';
import type { EventHandlerParams } from '@librechat/chat/react';
import type { EventSubmission } from 'librechat-data-provider';
import type { TResData, TFinalResData } from '~/common';
import { countTrailingOutputChars, markStreamStartFailedMetadata } from '~/utils';
import { markStreamStartFailedMetadata } from '~/utils';
import { reopenTermsOnRefusal } from '~/utils/terms';
import useLocalize from '~/hooks/useLocalize';

Expand Down Expand Up @@ -93,17 +98,8 @@ export default function useResumableEvents({
[rawStepHandler, run],
);

const {
bindResponse,
contextHandler,
usageHandler,
tapStream,
tapContent,
finalizeUsage,
backfillUsage,
resetLive,
seedLive,
} = useUsageHandler();
const usageHandlers = useUsageHandler();
const { resetLive } = usageHandlers;

const events = useMemo<ResumableEvents>(
() => ({
Expand Down Expand Up @@ -158,29 +154,7 @@ export default function useResumableEvents({
],
);

const usage = useMemo<ResumableUsage>(
() => ({
bindResponse,
contextHandler,
usageHandler,
tapStream,
tapContent,
finalizeUsage,
backfillUsage,
seedLiveFromContent: (content, usageSubmission) =>
seedLive(countTrailingOutputChars(content), usageSubmission),
}),
[
bindResponse,
contextHandler,
usageHandler,
tapStream,
tapContent,
finalizeUsage,
backfillUsage,
seedLive,
],
);
const usage = useMemo<ResumableUsage>(() => toResumableUsage(usageHandlers), [usageHandlers]);

return { events, usage };
}
66 changes: 66 additions & 0 deletions e2e/specs/mock/scenarios/gauge-follows-reply.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import { expect, test } from '@playwright/test';
import {
replyText,
replyPrompt,
messagesView,
NEW_CHAT_PATH,
MOCK_ENDPOINTS,
selectMockEndpoint,
sendMessageAndWaitForCompletion,
} from '../helpers';

/**
* The context gauge counts a reply as it is written: the reading the server sends when the
* reply starts, plus the output that has arrived since.
*/
test.use({ viewport: { width: 1440, height: 900 }, isMobile: false, hasTouch: false });

const uniqueLabel = (prefix: string) =>
`${prefix}-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;

/** The gauge's used-token count, read from its accessible name. */
const used = (label: string | null): number => {
const match = /Context window: ([\d.,]+)(K?) of/.exec(label ?? '');
if (match == null) {
throw new Error(`The gauge does not say how much is used: ${label}`);
}
return Number(match[1].replace(/,/g, '')) * (match[2] === 'K' ? 1000 : 1);
};

test('the gauge keeps counting while a reply streams @scenario:context-gauge-follows-a-streaming-reply', async ({
page,
}) => {
test.setTimeout(120_000);
const label = uniqueLabel('gauge');
await page.goto(NEW_CHAT_PATH, { timeout: 10_000 });
await selectMockEndpoint(page, MOCK_ENDPOINTS[0]);
const first = await sendMessageAndWaitForCompletion(page, replyPrompt(label));
expect(first.ok()).toBeTruthy();
await expect(messagesView(page).getByText(replyText(label))).toBeVisible();

const form = page.locator('main form').first();
const gauge = form.getByTestId('token-usage');
await form.getByRole('textbox', { name: 'Message input' }).click();
await page.keyboard.type(`E2E_SLOW_REPLY:${label}`);
await page.keyboard.press('Enter');

/**
* Early in the reply, and again well into it, both while it is still being written: more has
* arrived, so more is used. Read as they stand, with no waiting that could run past the end
* of the reply, where the count settles whatever it did meanwhile.
*/
const stop = page.getByRole('button', { name: 'Stop generating' });
await expect(messagesView(page)).toContainText('chunk-020', { timeout: 30_000 });
const early = used(await gauge.getAttribute('aria-label'));
await expect(messagesView(page)).toContainText('chunk-100', { timeout: 60_000 });
const late = used(await gauge.getAttribute('aria-label'));
await expect(stop).toBeVisible();
expect(late).toBeGreaterThan(early);

/** The settled count is not below what it showed on the way. */
await expect(messagesView(page)).toContainText('chunk-159', { timeout: 60_000 });
await expect(stop).toHaveCount(0);
await expect
.poll(async () => used(await gauge.getAttribute('aria-label')), { timeout: 10_000 })
.toBeGreaterThanOrEqual(late - 5);
});
34 changes: 34 additions & 0 deletions e2e/specs/mock/scenarios/side-chat-composer.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -375,3 +375,37 @@ for (const saveDrafts of [true, false]) {
await expect(main.getByText('comes-along.txt')).toHaveCount(1);
});
}

/** The gauge's used-token count, read from its accessible name. */
const usedTokens = (label: string | null): number => {
const match = /Context window: ([\d.,]+)(K?) of/.exec(label ?? '');
if (match == null) {
throw new Error(`The gauge does not say how much is used: ${label}`);
}
return Number(match[1].replace(/,/g, '')) * (match[2] === 'K' ? 1000 : 1);
};

test('the gauge beside the chat counts a reply as it is written @scenario:side-chat-composer-gauge-follows-the-reply', async ({
page,
}) => {
test.setTimeout(120_000);
const { sideChat, side, field } = await openBeside(page, 'gauge');
const gauge = side.getByTestId('token-usage');

await field.click();
await page.keyboard.type(`E2E_SLOW_REPLY:${uniqueLabel('measured')}`);
await page.keyboard.press('Enter');

/**
* Two readings while the reply is still running, as main chat's gauge gives them: the count
* the server sends as the reply starts, plus what has arrived since. No waiting between that
* could run past the end of the reply, where the count settles whatever it did meanwhile.
*/
const stop = side.getByRole('button', { name: 'Stop generating' });
await expect(sideChat).toContainText('chunk-020', { timeout: 30_000 });
const early = usedTokens(await gauge.getAttribute('aria-label'));
await expect(sideChat).toContainText('chunk-100', { timeout: 60_000 });
const late = usedTokens(await gauge.getAttribute('aria-label'));
await expect(stop).toBeVisible();
expect(late).toBeGreaterThan(early);
});
41 changes: 41 additions & 0 deletions packages/chat/src/core/__tests__/nameStepResponse.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import { Constants, StepEvents } from 'librechat-data-provider';
import type { TSubmission } from 'librechat-data-provider';
import type { StepEvent } from '../reducer';
import { nameStepResponse } from '../resumable/events';

const submission = { userMessage: { messageId: 'u1' } } as TSubmission;
const runStep = (runId: string | undefined): StepEvent =>
({ event: StepEvents.ON_RUN_STEP, data: { id: 's0', runId, index: 0 } }) as StepEvent;

describe('nameStepResponse', () => {
it('tells the usage writers the reply id a run step carries', () => {
const nameResponse = jest.fn();

nameStepResponse({ nameResponse }, runStep('stored-reply'), submission);

expect(nameResponse).toHaveBeenCalledWith('stored-reply', submission);
});

it('says nothing for a step that names no row of its own, or for any other event', () => {
const nameResponse = jest.fn();

nameStepResponse({ nameResponse }, runStep(undefined), submission);
nameStepResponse(
{ nameResponse },
runStep(Constants.USE_PRELIM_RESPONSE_MESSAGE_ID as string),
submission,
);
nameStepResponse(
{ nameResponse },
{ event: StepEvents.ON_MESSAGE_DELTA, data: { id: 's0', delta: {} } } as StepEvent,
submission,
);

expect(nameResponse).not.toHaveBeenCalled();
});

it('is harmless where no usage is kept', () => {
expect(() => nameStepResponse(undefined, runStep('stored-reply'), submission)).not.toThrow();
expect(() => nameStepResponse({}, runStep('stored-reply'), submission)).not.toThrow();
});
});
3 changes: 2 additions & 1 deletion packages/chat/src/core/apply.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import {
placeReasoningLabel,
} from './parts/place';
import { buildResumeEventSubmission, buildSubmissionFromResumeState } from './resumable/resume';
import { getStepEventId, isOAuthStepEvent } from './resumable/events';
import { getStepEventId, isOAuthStepEvent, nameStepResponse } from './resumable/events';
import { buildCreatedInitialResponse } from './resumable/messages';
import { placeResumedResponse } from './resumable/placement';
import { normalizeFrame } from './transport/frames';
Expand Down Expand Up @@ -520,6 +520,7 @@ export function createEventApplier(init: EventApplierInit = {}): EventApplier {
if (isOutputDelta(event)) {
init.usage?.tapStream(event.data as { delta?: { content?: unknown } }, submission);
}
nameStepResponse(init.usage, event, submission);
settle(reduceStep(stream, event, stepContext(submission, context)));
};

Expand Down
24 changes: 24 additions & 0 deletions packages/chat/src/core/resumable/events.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
import { Constants, StepEvents } from 'librechat-data-provider';
import type { TSubmission } from 'librechat-data-provider';
import type { ResumableUsage } from './host';
import type { StepEvent } from '../reducer';

const getToolCallName = (toolCall: unknown): unknown =>
toolCall != null && typeof toolCall === 'object' && 'name' in toolCall
Expand Down Expand Up @@ -74,3 +77,24 @@ export const getStepEventId = (event: unknown): string | undefined => {
? result.id
: undefined;
};

/**
* Tells the usage writers the id a run step gives the turn's reply. The reply is drawn under a
* placeholder id until then, and usage is bound to the submission that still names the
* placeholder; without this the reply being written stops counting as the one in progress the
* moment its row is renamed, and what a reader sees of its usage stands still until it ends.
*/
export function nameStepResponse(
usage: Pick<ResumableUsage, 'nameResponse'> | undefined,
event: StepEvent,
submission: TSubmission,
): void {
if (event.event !== StepEvents.ON_RUN_STEP) {
return;
}
const { runId } = event.data;
if (!runId || runId === Constants.USE_PRELIM_RESPONSE_MESSAGE_ID) {
return;
}
usage?.nameResponse?.(runId, submission);
}
Loading
Loading