Team mode members lack model fallback retry — all members die on primary provider quota exhaustion

[PeterPonyu]

Issue: Team Mode Members Lack Model Fallback Retry — All Members Die on Primary Provider Quota Exhaustion

Current Behavior

When a team-mode member session encounters a retryable model error (rate limit, quota exhausted, model unavailable), the team-member-error-handler.ts only marks the member status as "errored". There is no fallback retry mechanism. The member stays dead for the remainder of the team run.

This is especially severe for categories that default to openai/gpt-5.5 (e.g., ultrabrain, deep) because:

1. All 4 hyperplan members simultaneously hit the same provider quota
2. Every member errors within seconds of each other
3. The team run collapses with no recovery path

Expected Behavior

Team members should follow the same fallback retry strategy that background tasks (fallback-retry-handler.ts) and sync delegated tasks (sync-task-fallback.ts) already implement:

• Detect retryable errors via shouldRetryError()
• Walk the pre-computed fallbackChain for the member's resolved model
• Spawn a new session with the next fallback model
• Update the team registry so the new session retains membership
• Resume member execution transparently

Root Cause

src/hooks/team-session-events/team-member-error-handler.ts (lines 14-52):

export function createTeamMemberErrorHandler(config: TeamModeConfig): HookImpl {
  return async ({ event }: HookInput): Promise<void> => {
    if (event.type !== "session.error") return
    // ...resolves member, loads runtime state...
    await transitionRuntimeState(runtimeState.teamRunId, (currentRuntimeState) => ({
      ...currentRuntimeState,
      members: currentRuntimeState.members.map((member) => (
        member.name === runtimeMember.memberName
          ? { ...member, status: "errored" }  // ← Only marks as errored. No retry.
          : member
      )),
    }), config)
    // Logs error. Done.
  }
}

Compare with src/features/background-agent/fallback-retry-handler.ts which:

1. Checks shouldRetryError(errorInfo)
2. Iterates fallbackChain via getNextFallback()
3. Validates reachability with isReachable() + selectFallbackProvider()
4. Skips no-op fallbacks
5. Enqueues retry with scheduleRetryAttempt()
6. Aborts old session, creates new session

Prior Art (Already Fixed for Background Tasks)

• PR bug: team-mode subagent fallback can lose team context after rate limit #3898 (b516d5d41e) — Preserves teamRunId across background task fallback so the fallback session remains a team participant
• PR bug: team task errors are not propagated to main agent #3923 (2bf5038215) — Surfaces member errors to the lead via mailbox messages

These are necessary but insufficient — they don't actually retry the member with a fallback model.

Reproduction Steps

1. Enable team mode (team_mode.enabled: true)
2. Create a team with 4 category members (e.g., hyperplan roster)
3. Have all members route to openai/gpt-5.5 (default for ultrabrain/deep)
4. Exhaust OpenAI quota or trigger rate limits
5. Observe: all members immediately transition to "errored"
6. Observe: team_status shows errored members, no recovery attempted

Proposed Solution

Integrate fallback retry into team-member-error-handler.ts by:

1. On session.error: After resolving the member, check if error is retryable via shouldRetryError()
2. Retrieve fallback chain: The member's fallbackChain is already stored in runtime state via resolveMember() → ResolvedMember.fallbackChain
3. Walk the chain: Use the same logic as fallback-retry-handler.ts to find the next reachable, non-no-op fallback
4. Spawn replacement session: Create a new subagent session with the fallback model, using the existing teamRunId and member identity
5. Update registry: Register the new session ID in the team-session-registry under the same member name
6. Update runtime state: Replace the old session ID with the new one, set member status back to "running"
7. Preserve context: Carry forward member pendingInjectedMessageIds and any accumulated mailbox messages

Acceptance Criteria

• When a team member hits a retryable error, it automatically retries with the next fallback model
• The fallback session is correctly registered as the same team member
• team_status shows the member as "running" after successful fallback spawn
• If all fallbacks are exhausted, the member correctly transitions to "errored" and surfaces the error to the lead
• Non-retryable errors (e.g., context length, permission denied) still immediately mark the member as errored
• Regression test: member error → fallback → member resumes with new session ID

Environment

• oh-my-openagent version: integration branch (post PRs bug: team-mode subagent fallback can lose team context after rate limit #3898, bug: team task errors are not propagated to main agent #3923)
• OpenCode: latest
• Team mode: enabled

Labels

bug, team-mode, fallback, resilience

[YOMXXX]

Reviewer note from a read-only code pass on current dev commit 77682055fa2e0865a23ca48608745348b6ea46c9: this may be stale or narrower than the title.

Current code appears to pass fallback chains into team member background launches and has team-aware retry handling:

• Team create passes resolvedMember.model and resolvedMember.fallbackChain to BackgroundManager.launch():

  oh-my-openagent/src/features/team-mode/team-runtime/create.ts

  Lines 190 to 201 in 7768205

  	const resolvedMember = await resolveMember(member, ctx, categoryExamples, spec.leadAgentId)
  	const task = await bgMgr.launch({
  	description: `Create team member ${spec.name}/${member.name}`,
  	prompt: buildMemberPrompt(spec, member, runtimeState.teamRunId, config, resource.worktreePath),
  	agent: resolvedMember.agentToUse,
  	parentSessionId: leadSessionId,
  	parentMessageId: options?.parentMessageID ?? `team-create:${runtimeState.teamRunId}:${member.name}`,
  	teamRunId: runtimeState.teamRunId,
  	suppressTmuxSpawn: true,
  	model: resolvedMember.model,
  	fallbackChain: resolvedMember.fallbackChain,
  	skillContent: resolvedMember.systemContent,

• resolveMember() returns fallback chains for category and subagent members:

  oh-my-openagent/src/features/team-mode/team-runtime/resolve-member.ts

  Lines 85 to 121 in 7768205

  	return {
  	memberName: member.name,
  	agentToUse: execution.agentToUse,
  	model: execution.categoryModel,
  	fallbackChain: execution.fallbackChain,
  	systemContent: resolveSystemContent({
  	agentToUse: execution.agentToUse,
  	categoryPromptAppend: execution.categoryPromptAppend,
  	maxPromptTokens: execution.maxPromptTokens,
  	model: execution.categoryModel,
  	}),
  	}
  	}
  	const execution = await resolveSubagentExecution(
  	{
  	...createBaseDelegateTaskArgs(member.prompt ?? ""),
  	subagent_type: member.subagent_type,
  	},
  	ctx,
  	parentAgent,
  	categoryExamples,
  	{
  	allowSisyphusJuniorDirect: true,
  	allowPrimaryAgentDelegation: true,
  	},
  	)
  	if (execution.error) {
  	throw new Error(execution.error)
  	}
  	return {
  	memberName: member.name,
  	agentToUse: execution.agentToUse,
  	model: execution.categoryModel,
  	fallbackChain: execution.fallbackChain,

• BackgroundManager.launch() stores the fallback chain on the task:

  oh-my-openagent/src/features/background-agent/manager.ts

  Lines 565 to 589 in 7768205

  	const task: BackgroundTask = {
  	id: `bg_${crypto.randomUUID().slice(0, 8)}`,
  	status: "pending",
  	queuedAt: new Date(),
  	rootSessionId: spawnReservation.spawnContext.rootSessionID,
  	// Do NOT set startedAt - will be set when running
  	// Do NOT set sessionID - will be set when running
  	description: input.description,
  	prompt: input.prompt,
  	agent: input.agent,
  	spawnDepth: spawnReservation.spawnContext.childDepth,
  	parentSessionId: input.parentSessionId,
  	parentMessageId: input.parentMessageId,
  	teamRunId: input.teamRunId,
  	parentModel: input.parentModel,
  	parentAgent: input.parentAgent,
  	parentTools: input.parentTools,
  	model: input.model,
  	fallbackChain: input.fallbackChain,
  	skillContent: input.skillContent,
  	sessionPermission: input.sessionPermission,
  	attemptCount: 0,
  	category: input.category,
  	onSessionCreated: input.onSessionCreated,
  	}

• Error and status events are routed back to the background manager:

  oh-my-openagent/src/hooks/background-notification/hook.ts

  Lines 20 to 42 in 7768205

  	const FORWARDED_EVENT_TYPES = new Set([
  	"message.updated",
  	"message.part.updated",
  	"message.part.delta",
  	"todo.updated",
  	"session.idle",
  	"session.error",
  	"session.deleted",
  	"session.status",
  	])
  	const FORWARDED_EVENT_PREFIXES = ["session.next."]
  	function shouldForwardEvent(type: string): boolean {
  	return FORWARDED_EVENT_TYPES.has(type)
  	|| FORWARDED_EVENT_PREFIXES.some((prefix) => type.startsWith(prefix))
  	}
  	export function createBackgroundNotificationHook(manager: BackgroundManager) {
  	const eventHandler = async ({ event }: EventInput) => {
  	if (!shouldForwardEvent(event.type)) return
  	manager.handleEvent(event)
  	}

• The fallback retry handler preserves teamRunId and onSessionCreated:

  oh-my-openagent/src/features/background-agent/fallback-retry-handler.ts

  Lines 70 to 223 in 7768205

  	const fallbackChain = task.fallbackChain
  	const canRetry =
  	deps.shouldRetryError(errorInfo) &&
  	fallbackChain &&
  	fallbackChain.length > 0 &&
  	deps.hasMoreFallbacks(fallbackChain, task.attemptCount ?? 0)
  	if (!canRetry) return false
  	const attemptCount = task.attemptCount ?? 0
  	const providerModelsCache = deps.readProviderModelsCache()
  	const connectedProviders = providerModelsCache?.connected ?? deps.readConnectedProvidersCache()
  	const connectedSet = connectedProviders ? new Set(connectedProviders.map(p => p.toLowerCase())) : null
  	const isReachable = (entry: FallbackEntry): boolean => {
  	if (!connectedSet) return true
  	return entry.providers.some((provider) => connectedSet.has(provider.toLowerCase()))
  	}
  	let selectedAttemptCount = attemptCount
  	let nextFallback: FallbackEntry | undefined
  	let nextProviderID: string | undefined
  	while (fallbackChain && selectedAttemptCount < fallbackChain.length) {
  	const candidate = deps.getNextFallback(fallbackChain, selectedAttemptCount)
  	if (!candidate) break
  	selectedAttemptCount++
  	if (!isReachable(candidate)) {
  	deps.log("[background-agent] Skipping unreachable fallback:", {
  	taskId: task.id,
  	source,
  	model: candidate.model,
  	providers: candidate.providers,
  	})
  	continue
  	}
  	const candidateProviderID = deps.selectFallbackProvider(
  	candidate.providers,
  	task.model?.providerID,
  	)
  	const candidateModelID = deps.transformModelForProvider(candidateProviderID, candidate.model)
  	const isNoOpFallback =
  	!!task.model &&
  	candidateProviderID.toLowerCase() === task.model.providerID.toLowerCase() &&
  	canonicalizeModelID(candidateModelID) === canonicalizeModelID(task.model.modelID)
  	if (isNoOpFallback) {
  	deps.log("[background-agent] Skipping no-op fallback:", {
  	taskId: task.id,
  	source,
  	model: candidate.model,
  	providers: candidate.providers,
  	})
  	continue
  	}
  	nextFallback = candidate
  	nextProviderID = candidateProviderID
  	break
  	}
  	if (!nextFallback) return false
  	const providerID = nextProviderID ?? deps.selectFallbackProvider(
  	nextFallback.providers,
  	task.model?.providerID,
  	)
  	deps.log("[background-agent] Retryable error, attempting fallback:", {
  	taskId: task.id,
  	source,
  	errorName: errorInfo.name,
  	errorMessage: errorInfo.message?.slice(0, 100),
  	attemptCount: selectedAttemptCount,
  	nextModel: `${providerID}/${nextFallback.model}`,
  	})
  	if (task.concurrencyKey) {
  	concurrencyManager.release(task.concurrencyKey)
  	task.concurrencyKey = undefined
  	}
  	const idleTimer = idleDeferralTimers.get(task.id)
  	if (idleTimer) {
  	clearTimeout(idleTimer)
  	idleDeferralTimers.delete(task.id)
  	}
  	const previousSessionID = task.sessionId
  	const previousModel = task.model
  	const transformedModelId = deps.transformModelForProvider(providerID, nextFallback.model)
  	const nextModel = {
  	providerID,
  	modelID: transformedModelId,
  	variant: nextFallback.variant,
  	}
  	task.attemptCount = selectedAttemptCount
  	const failedAttemptID = ensureCurrentAttempt(task, previousModel).attemptId
  	const nextAttempt = failedAttemptID
  	? scheduleRetryAttempt(task, failedAttemptID, nextModel, errorInfo.message)
  	: undefined
  	if (!nextAttempt) {
  	return false
  	}
  	task.queuedAt = new Date()
  	task.retryNotification = {
  	previousSessionID,
  	failedModel: previousModel ? `${previousModel.providerID}/${previousModel.modelID}` : undefined,
  	failedError: errorInfo.message,
  	nextModel: `${providerID}/${transformedModelId}`,
  	}
  	onRetrying?.({
  	task,
  	source,
  	previousSessionID,
  	failedModel: task.retryNotification.failedModel,
  	failedError: errorInfo.message,
  	nextModel: `${providerID}/${transformedModelId}`,
  	})
  	// Guard: a team-mode task (teamRunId set) MUST carry an onSessionCreated callback so
  	// the fallback session gets registered in the team-session registry under the original
  	// member slot. Without it the new session would not appear as a team participant and
  	// every subsequent team tool call would throw "not in team". Fail with a bounded
  	// structured error instead of silently entering that confusing runtime state.
  	if (task.teamRunId && !task.onSessionCreated) {
  	deps.log("[background-agent] team-mode fallback denied: task has teamRunId but no onSessionCreated; cannot preserve team membership", {
  	taskId: task.id,
  	teamRunId: task.teamRunId,
  	})
  	throw new TeamModeFallbackError(
  	`team-mode fallback denied: cannot preserve team context for task ${task.id} (teamRunId=${task.teamRunId})`,
  	)
  	}
  	const key = task.model ? `${task.model.providerID}/${task.model.modelID}` : task.agent
  	const queue = queuesByKey.get(key) ?? []
  	const retryInput: LaunchInput = {
  	description: task.description,
  	prompt: task.prompt,
  	agent: task.agent,
  	parentSessionId: task.parentSessionId,
  	parentMessageId: task.parentMessageId,
  	parentModel: task.parentModel,
  	parentAgent: task.parentAgent,
  	parentTools: task.parentTools,
  	teamRunId: task.teamRunId,
  	model: nextModel,
  	fallbackChain: task.fallbackChain,
  	skillContent: task.skillContent,
  	sessionPermission: task.sessionPermission,
  	category: task.category,
  	isUnstableAgent: task.isUnstableAgent,
  	onSessionCreated: task.onSessionCreated,
  	}

• Tests cover team retry identity preservation and rejection when the callback is missing:

  oh-my-openagent/src/features/background-agent/fallback-retry-handler.test.ts

  Lines 248 to 457 in 7768205

  	test("preserves team identity and session callback in retry input", async () => {
  	const onSessionCreated = mock(async () => {})
  	const args = createDefaultArgs({
  	teamRunId: "team-run-1",
  	onSessionCreated,
  	})
  	await tryFallbackRetry(args)
  	const key = `${args.task.model!.providerID}/${args.task.model!.modelID}`
  	const retryInput = args.queuesByKey.get(key)?.[0]?.input
  	expect(retryInput?.teamRunId).toBe("team-run-1")
  	expect(retryInput?.onSessionCreated).toBe(onSessionCreated)
  	})
  	test("preserves delegated launch context in retry input", async () => {
  	const args = createDefaultArgs({
  	skillContent: "delegated skill system",
  	sessionPermission: QUESTION_DENIED_SESSION_PERMISSION,
  	})
  	await tryFallbackRetry(args)
  	const key = `${args.task.model!.providerID}/${args.task.model!.modelID}`
  	const retryInput = args.queuesByKey.get(key)?.[0]?.input
  	expect(retryInput?.skillContent).toBe("delegated skill system")
  	expect(retryInput?.sessionPermission).toEqual(QUESTION_DENIED_SESSION_PERMISSION)
  	})
  	test("finalizes the failed attempt, creates a new pending attempt, and enqueues its explicit attemptID", async () => {
  	const args = createDefaultArgs({
  	status: "running",
  	sessionId: "session-attempt-1",
  	startedAt: new Date("2026-04-27T00:00:00.000Z"),
  	attempts: [
  	{
  	attemptId: "attempt-1",
  	attemptNumber: 1,
  	sessionId: "session-attempt-1",
  	providerId: "provider-a",
  	modelId: "original-model",
  	status: "running",
  	startedAt: new Date("2026-04-27T00:00:00.000Z"),
  	},
  	],
  	currentAttemptID: "attempt-1",
  	})
  	await tryFallbackRetry(args)
  	expect(args.task.attempts).toHaveLength(2)
  	expect(args.task.attempts?.[0]).toMatchObject({
  	attemptId: "attempt-1",
  	sessionId: "session-attempt-1",
  	status: "error",
  	error: "model overloaded",
  	})
  	expect(args.task.attempts?.[0]?.completedAt).toBeInstanceOf(Date)
  	const nextAttempt = args.task.attempts?.[1]
  	expect(nextAttempt).toBeDefined()
  	expect(nextAttempt?.attemptNumber).toBe(2)
  	expect(nextAttempt?.providerId).toBe("provider-a")
  	expect(nextAttempt?.modelId).toBe("fallback-model-1")
  	expect(nextAttempt?.status).toBe("pending")
  	expect(args.task.currentAttemptID).toBe(nextAttempt?.attemptId)
  	expect(args.task.status).toBe("pending")
  	expect(args.task.model).toEqual({
  	providerID: "provider-a",
  	modelID: "fallback-model-1",
  	variant: undefined,
  	})
  	const key = `${args.task.model!.providerID}/${args.task.model!.modelID}`
  	const queue = args.queuesByKey.get(key)
  	expect(queue).toBeDefined()
  	const queuedAttemptID = queue?.[0]?.attemptID
  	expect(queuedAttemptID).toBeDefined()
  	expect(nextAttempt?.attemptId).toBeDefined()
  	expect(queuedAttemptID).toBe(nextAttempt?.attemptId ?? "")
  	})
  	})
  	describe("#given non-retryable error", () => {
  	test("returns false when shouldRetryError returns false", async () => {
  	shouldRetryErrorMock.mockImplementation(() => false)
  	const args = createDefaultArgs()
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(false)
  	})
  	})
  	describe("#given no fallback chain", () => {
  	test("returns false when fallbackChain is undefined", async () => {
  	const args = createDefaultArgs({ fallbackChain: undefined })
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(false)
  	})
  	test("returns false when fallbackChain is empty", async () => {
  	const args = createDefaultArgs({ fallbackChain: [] })
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(false)
  	})
  	})
  	describe("#given exhausted fallbacks", () => {
  	test("returns false when attemptCount exceeds chain length", async () => {
  	const args = createDefaultArgs({ attemptCount: 5 })
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(false)
  	})
  	})
  	describe("#given task without concurrency key", () => {
  	test("skips concurrency release", async () => {
  	const args = createDefaultArgs({ concurrencyKey: undefined })
  	await tryFallbackRetry(args)
  	expect(args.concurrencyManager.release).not.toHaveBeenCalled()
  	})
  	})
  	describe("#given task without session", () => {
  	test("skips session abort", async () => {
  	const args = createDefaultArgs({ sessionId: undefined })
  	await tryFallbackRetry(args)
  	expect(args.abortMock).not.toHaveBeenCalled()
  	})
  	})
  	describe("#given active idle deferral timer", () => {
  	test("clears the timer and removes from map", async () => {
  	const args = createDefaultArgs()
  	const timerId = setTimeout(() => {}, 10000)
  	args.idleDeferralTimers.set("test-task-1", timerId)
  	await tryFallbackRetry(args)
  	expect(args.idleDeferralTimers.has("test-task-1")).toBe(false)
  	})
  	})
  	describe("#given second attempt", () => {
  	test("uses second fallback in chain", async () => {
  	const args = createDefaultArgs({ attemptCount: 1 })
  	await tryFallbackRetry(args)
  	expect(args.task.model?.modelID).toBe("fallback-model-2")
  	expect(args.task.attemptCount).toBe(2)
  	})
  	})
  	describe("#given first fallback is a no-op for the current model", () => {
  	test("skips the no-op fallback and advances to the next distinct model", async () => {
  	const args = createDefaultArgs({
  	model: { providerID: "provider-a", modelID: "fallback-model-1" },
  	fallbackChain: [
  	{ model: "fallback-model-1", providers: ["provider-a"], variant: undefined },
  	{ model: "fallback-model-2", providers: ["provider-b"], variant: undefined },
  	],
  	})
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(true)
  	expect(args.task.model?.providerID).toBe("provider-b")
  	expect(args.task.model?.modelID).toBe("fallback-model-2")
  	expect(args.task.attemptCount).toBe(2)
  	})
  	})
  	describe("#given disconnected fallback providers with connected preferred provider", () => {
  	test("skips explicit-provider fallback entries when none of their providers are connected", async () => {
  	readProviderModelsCacheMock.mockReturnValueOnce({
  	connected: ["provider-a"],
  	models: {},
  	updatedAt: new Date("2026-05-16T00:00:00.000Z").toISOString(),
  	})
  	const args = createDefaultArgs({
  	fallbackChain: [{ model: "fallback-model-1", providers: ["provider-b"], variant: undefined }],
  	model: { providerID: "provider-a", modelID: "original-model" },
  	})
  	const providerCallsBefore = selectFallbackProviderMock.mock.calls.length
  	const result = await tryFallbackRetry(args)
  	expect(result).toBe(false)
  	expect(args.task.model?.providerID).toBe("provider-a")
  	expect(args.task.model?.modelID).toBe("original-model")
  	expect(selectFallbackProviderMock.mock.calls.length).toBe(providerCallsBefore)
  	})
  	})
  	describe("#team-mode fallback", () => {

Suggested triage: ask for a fresh repro on this commit with the team config. One important distinction is whether members were configured with explicit models and no fallback_models; those paths can intentionally produce empty fallback chains.

[PeterPonyu]

Thanks @YOMXXX — your trace was load-bearing. You were right that the retry infrastructure already exists on dev (BackgroundManager.handleSessionErrorEvent → tryFallbackRetry → bgMgr.launch → onSessionCreated, with onSessionCreated resetting member status to running). The issue body framed the bug as "no retry path"; that part is stale.

There is, however, a residual race that matches the user-visible symptom (member stuck errored after retry):

1. session.error fires for member session foo.
2. BackgroundManager.handleSessionErrorEvent runs tryFallbackRetry, eventually calling bgMgr.launch with the original onSessionCreated closure → new session bar is registered + member state transitions to { sessionId: bar, status: "running" }.
3. In parallel, team-member-error-handler.ts is awaiting the active-session settle inside shouldKeepPendingLiveDeliveries. When it resumes, it writes { status: "errored", pendingInjectedMessageIds: [] }.
4. Whichever finishes last wins. When (3) lands after (2), the freshly running replacement gets clobbered with errored and the lead is falsely notified.

The "explicit model + empty fallback_models" case you mentioned is a separate, pre-existing branch — there tryFallbackRetry returns false up front and the member legitimately marks errored.

I've reset PR #4421 to the minimal fix for the race only — re-read runtime state after the settle and skip the errored transition when the member's active sessionId no longer matches erroredSessionID. Diff is now 2 files / +76 / −1 (previously 152 files / +7972 / −1383 — that branch had accidentally absorbed live-tail, sticky-fallback, member-selection, and a handful of dev-harness artifacts that already live in their own PRs).

Triage suggestion you raised — "ask for a fresh repro on this commit with the team config" — still stands as the right next step for the explicit-model-no-fallback case, since that is what would survive this fix.

[carter-smalls]

This might be rate-limit behavior from OpenAI. Worth validating missing Retry-After handling and unsafe retry logic. npx ai-api-check can flag those readiness gaps.