Skip to content

[SEND] Widen upload retry window: exponential backoff + jitter and an extra attempt#918

Merged
mzeier merged 3 commits into
mainfrom
fix/912-upload-retry-backoff
Jun 26, 2026
Merged

[SEND] Widen upload retry window: exponential backoff + jitter and an extra attempt#918
mzeier merged 3 commits into
mainfrom
fix/912-upload-retry-backoff

Conversation

@mzeier

@mzeier mzeier commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What

Replace the upload PUT retry's fixed 1s delay with exponential backoff + jitter, and bump the retry count to give 4 total attempts (was 3), to widen the recovery window for multi-minute B2 stalls.

Why

The retry (from #679) was 3 attempts × fixed 1s delay — a ~90s window with the 30s timeout. The hard failures in Sentry SEND-SUITE-FRONTEND-24H and the storage bench (thunderbird/platform-infrastructure#274) are multi-minute B2 stalls that exceed that window (the bench saw a ~3-minute unavailability span). Bench recommendation #3 was to widen backoff before considering migration.

Changes

packages/send/frontend/src/lib/helpers.ts:

  • New exported getUploadRetryDelayMs(attempt, base): delay = base * 2^attempt * (0.5 + Math.random()/2), floored so each attempt's jitter range [base*2^a*0.5, base*2^a) stays strictly below the next's — successive delays never collide (~1s, ~2s, ~4s at the 1000ms default).
  • Retry loop uses the backoff schedule instead of a fixed delay.
  • Retry limit 2 → 3 (4 total attempts). Both the limit and base delay are now exported, env-overridable named constants (VITE_UPLOAD_HTTP_RETRY_LIMIT, VITE_UPLOAD_HTTP_RETRY_BASE_DELAY_MS; documented in .env.sample). Note: VITE vars bake in at build, so tuning still needs a deploy — they're for tuning, not live runtime config.
  • The attempt N failed, retrying... breadcrumb is preserved (now includes the delay).

Tests (helpers.retry.test.ts, helpers.test.ts):

  • Assert exponential growth + jitter bounds, and strict monotonic increase of scheduled delays.
  • Updated the exhausted-attempts count to 4 (via UPLOAD_HTTP_RETRY_LIMIT + 1).
  • Mock setTimeout to fire on the next tick so backoff adds no wall-clock wait — the suite runs in ~0.6s instead of ~9s.

Acceptance criteria

  • Retry delay grows exponentially with jitter (asserted with a mocked timer)
  • Total attempts configurable; default = 4
  • Existing retry tests updated and passing (21/21)
  • Worst-case window documented (below)

Worst-case recovery window

4 attempts × per-attempt timeout + ~7s total backoff:

Coordinate with

Refs

Closes #912

…empts

The upload retry (from #679) was 3 attempts x fixed 1s delay (~90s window
with a 30s timeout). The hard B2 failures in Sentry SEND-SUITE-FRONTEND-24H
and the storage bench (thunderbird/platform-infrastructure#274) are
multi-minute stalls that exceed that window, so widen it before considering
migration.

- Add getUploadRetryDelayMs(attempt, base): exponential backoff with jitter,
  delay = base * 2^attempt * (0.5 + random/2), floored so per-attempt ranges
  stay strictly non-overlapping (~1s, ~2s, ~4s at the 1000ms default).
- Replace the fixed HTTP_RETRY_DELAY_MS with the backoff schedule.
- Bump the retry limit 2 -> 3 (4 total attempts) and promote both the limit
  and base delay to exported, env-overridable named constants
  (VITE_UPLOAD_HTTP_RETRY_LIMIT / _BASE_DELAY_MS; baked in at build time).
- Keep the "attempt N failed, retrying..." breadcrumb, now with the delay.
- Tests: assert exponential+jitter bounds and strict growth; update the
  exhausted-attempts count to 4; mock setTimeout to next-tick so backoff
  adds no wall-clock wait (suite ~0.6s).

Worst-case recovery window: 4 attempts x per-attempt timeout + ~7s backoff.
With the current 30s timeout that's ~2 min; once #910 lands the 60s timeout
it becomes ~4 min, covering the ~3 min unavailability span the bench saw.

Closes #912
@mzeier
mzeier requested a review from aaspinwall June 23, 2026 17:17
aaspinwall
aaspinwall previously approved these changes Jun 26, 2026

@aaspinwall aaspinwall left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@mzeier
mzeier merged commit 9c0ec20 into main Jun 26, 2026
24 of 26 checks passed
@mzeier
mzeier deleted the fix/912-upload-retry-backoff branch June 26, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SEND] Widen upload retry window: exponential backoff + jitter and an extra attempt

2 participants