Skip to content

Let clients resubmit samples that failed on a registry read - #291

Merged
Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/registry-cold-read-retry
Nov 16, 2026
Merged

Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/registry-cold-read-retry

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A sample whose container fails to read the recipe registry is now reported to the client as lost (HTTP 410) instead of failed. The Tinker SDK resubmits a lost request, so a brief registry outage no longer ends the client's run. The frontend logs the registry error behind each lost sample, so a lasting outage stays visible.

When many sampling containers start at once, their first registry reads can exceed the storage list rate limit. Volume list calls already retry rate limits for about 15 s (_retry_rate_limited in providers/modal/kv.py). A burst that outlasts those retries made execute_sample raise RegistryUnavailable, which the control plane reported as a failed sample. Tinker clients don't retry a failed sample, so the run stopped. The HTTP API already returns 503 for the same error, which the SDK retries.

This replaces the earlier version of this PR, which added a second retry loop around the first registry read. Retrying is left to the existing Volume retry and the SDK's lost-request retry.

E2E results

Setup: stock tinker-cookbook 0.6.1 math_rl (gsm8k, 8 groups × 4 samples) against a staging deployment of a large LoRA recipe.

Run First sampling round
Before Samples failed with registry read failed: VolumeListFiles rate limit exceeded, and the client stopped. This happened on two fresh deployments.
After No registry errors; the run completed 5 RL steps.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cold sampling container that cannot read the recipe registry now reports
its sample as lost (410), which the Tinker SDK resubmits, instead of failed.
Drops the first-read retry loop: Volume list calls already retry rate limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Tinker SDK resubmits a lost (410) request, so a cold sampling container
that cannot read the recipe registry no longer ends the client's run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong Micah Yong (micahtyong) changed the title Retry a new container's first registry read Let clients resubmit samples that failed on a registry read Nov 15, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) merged commit 2edd6be into main Nov 16, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/registry-cold-read-retry branch November 16, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant