[SEND] Widen upload retry window: exponential backoff + jitter and an extra attempt#918
Merged
Conversation
…empts The upload retry (from #679) was 3 attempts x fixed 1s delay (~90s window with a 30s timeout). The hard B2 failures in Sentry SEND-SUITE-FRONTEND-24H and the storage bench (thunderbird/platform-infrastructure#274) are multi-minute stalls that exceed that window, so widen it before considering migration. - Add getUploadRetryDelayMs(attempt, base): exponential backoff with jitter, delay = base * 2^attempt * (0.5 + random/2), floored so per-attempt ranges stay strictly non-overlapping (~1s, ~2s, ~4s at the 1000ms default). - Replace the fixed HTTP_RETRY_DELAY_MS with the backoff schedule. - Bump the retry limit 2 -> 3 (4 total attempts) and promote both the limit and base delay to exported, env-overridable named constants (VITE_UPLOAD_HTTP_RETRY_LIMIT / _BASE_DELAY_MS; baked in at build time). - Keep the "attempt N failed, retrying..." breadcrumb, now with the delay. - Tests: assert exponential+jitter bounds and strict growth; update the exhausted-attempts count to 4; mock setTimeout to next-tick so backoff adds no wall-clock wait (suite ~0.6s). Worst-case recovery window: 4 attempts x per-attempt timeout + ~7s backoff. With the current 30s timeout that's ~2 min; once #910 lands the 60s timeout it becomes ~4 min, covering the ~3 min unavailability span the bench saw. Closes #912
…backoff # Conflicts: # packages/send/frontend/src/lib/helpers.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the upload PUT retry's fixed 1s delay with exponential backoff + jitter, and bump the retry count to give 4 total attempts (was 3), to widen the recovery window for multi-minute B2 stalls.
Why
The retry (from #679) was 3 attempts × fixed 1s delay — a ~90s window with the 30s timeout. The hard failures in Sentry SEND-SUITE-FRONTEND-24H and the storage bench (thunderbird/platform-infrastructure#274) are multi-minute B2 stalls that exceed that window (the bench saw a ~3-minute unavailability span). Bench recommendation #3 was to widen backoff before considering migration.
Changes
packages/send/frontend/src/lib/helpers.ts:getUploadRetryDelayMs(attempt, base):delay = base * 2^attempt * (0.5 + Math.random()/2), floored so each attempt's jitter range[base*2^a*0.5, base*2^a)stays strictly below the next's — successive delays never collide (~1s, ~2s, ~4s at the 1000ms default).2 → 3(4 total attempts). Both the limit and base delay are now exported, env-overridable named constants (VITE_UPLOAD_HTTP_RETRY_LIMIT,VITE_UPLOAD_HTTP_RETRY_BASE_DELAY_MS; documented in.env.sample). Note: VITE vars bake in at build, so tuning still needs a deploy — they're for tuning, not live runtime config.attempt N failed, retrying...breadcrumb is preserved (now includes the delay).Tests (
helpers.retry.test.ts,helpers.test.ts):UPLOAD_HTTP_RETRY_LIMIT + 1).setTimeoutto fire on the next tick so backoff adds no wall-clock wait — the suite runs in ~0.6s instead of ~9s.Acceptance criteria
Worst-case recovery window
4 attempts × per-attempt timeout + ~7s total backoff:
Coordinate with
helpers.ts(it does not touchXHR_TIMEOUT_MS), so the two won't conflict.Refs
Closes #912