Repository navigation
Let clients resubmit samples that failed on a registry read - #291
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
force-pushed
the
micah/registry-cold-read-retry
branch
from
November 13, 2026 13:53
11776af to
2a0f40e
Compare
Micah Yong (micahtyong)
marked this pull request as ready for review
November 15, 2026 22:40
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 15, 2026 22:40
A cold sampling container that cannot read the recipe registry now reports its sample as lost (410), which the Tinker SDK resubmits, instead of failed. Drops the first-read retry loop: Volume list calls already retry rate limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Tinker SDK resubmits a lost (410) request, so a cold sampling container that cannot read the recipe registry no longer ends the client's run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A sample whose container fails to read the recipe registry is now reported to the client as lost (HTTP 410) instead of failed. The Tinker SDK resubmits a lost request, so a brief registry outage no longer ends the client's run. The frontend logs the registry error behind each lost sample, so a lasting outage stays visible.
When many sampling containers start at once, their first registry reads can exceed the storage list rate limit. Volume list calls already retry rate limits for about 15 s (
_retry_rate_limitedinproviders/modal/kv.py). A burst that outlasts those retries madeexecute_sampleraiseRegistryUnavailable, which the control plane reported as a failed sample. Tinker clients don't retry a failed sample, so the run stopped. The HTTP API already returns 503 for the same error, which the SDK retries.This replaces the earlier version of this PR, which added a second retry loop around the first registry read. Retrying is left to the existing Volume retry and the SDK's lost-request retry.
E2E results
Setup: stock tinker-cookbook 0.6.1
math_rl(gsm8k, 8 groups × 4 samples) against a staging deployment of a large LoRA recipe.registry read failed: VolumeListFiles rate limit exceeded, and the client stopped. This happened on two fresh deployments.🤖 Generated with Claude Code