Skip to content

feat: streaming upload finalize + periodic session GC - #14

Merged
deckyfx merged 5 commits into
mainfrom
feat/upload-gc-and-streaming-finalize
Jun 30, 2026
Merged

deckyfx merged 5 commits into
mainfrom
feat/upload-gc-and-streaming-finalize

Conversation

@deckyfx

@deckyfx deckyfx commented Jun 30, 2026 •

Copy link
Copy Markdown
Owner

Two follow-ups noted in the resumable-uploads PR (#13), shipped together.

Streaming finalize

The finalize path no longer reads the whole staged file (up to the multi-GB cap) into memory.

  • staging exposes open(key): Blob (a lazy Bun.file) instead of readAll.
  • asset-service grows createFromSource(blob, meta) + an internal ingest pipeline shared with create({ bytes }):
    • stream-hashes sha256 + md5 in one pass over the blob,
    • sniffs via Bun.Image(blob) (reads it lazily; the maxPixels decompression-bomb guard is intact),
    • stores the blob itself.
      create wraps ingest with an in-memory Blob snapshot, so POST /assets and the seeder are unchanged.
  • Upload finalize streams from staging.open(stagingKey); the per-session lock, cleanup ordering, and keep-on-transient behaviour are untouched.
  • storage.store now also accepts a Blob and hands it straight to Bun.write. Wrapping a file-backed stream in a Response stalls Bun.write (a real hang caught by the integration test), so callers pass the Blob, not blob.stream(). That store branch was never exercised before — create only ever passed a Uint8Array.

A Bun.Image spike confirmed it accepts a Blob/path lazily and that streaming sha256/md5 are bit-identical to full-buffer hashing.

Periodic GC

Expired upload sessions + their staging files were only reclaimed opportunistically on the next upload. The API composition root now drives uploadService.gcExpired() on a timer: UPLOAD_GC_INTERVAL_MS (default 15 min, 0 disables, capped at the 32-bit timer ceiling), error-isolated, unref()'d, cleared on shutdown.

Tests

  • asset-service streaming-equivalence (in-memory vs. source → identical hashes/metadata/key), dedupe, and undecodable-source rejection.
  • gcExpired sweep + no-op; env-config GC bounds (default/parse/0/negative/over-cap).
  • A real-filesystem finalize integration test: positional chunks → file-backed streaming hash/sniff/store → byte round-trip (also guards the Bun.write(Blob) path).

All green: typecheck · 154 tests · lint:boundaries. CodeRabbit CLI pre-review applied (timer-ceiling cap); one finding (delete orphan blob on insert failure) intentionally skipped — see below.

Out of scope / follow-ups

  • Orphan-GC for content-addressed blobs. A non-race insert failure can leave a stored blob unreferenced. Inline deletion is unsafe here (the key is content-addressed; a concurrent same-content upload may own it — deleting would orphan its row), so this needs a separate reference-counted sweep. Documented in code.
  • Raising MAX_UPLOAD_BYTES now that RAM is no longer the finalize ceiling (policy call).
  • A staging→final move() fast-path for the FS provider (can't assume FS since the store may be S3).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added configurable background cleanup for expired uploads via UPLOAD_GC_INTERVAL_MS (defaulting to 15 minutes, with 0 to disable).
    • Improved asset ingest during upload finalization to use streaming from staged data to reduce memory usage.
    • Expanded storage support to accept Blob inputs.
  • Bug Fixes
    • Made cleanup safer and more reliable by processing expired items in bounded batches and avoiding interference with in-flight finalization.
    • Added stricter validation for interval configuration, including an upper delay ceiling.
  • Tests
    • Added/extended coverage for streaming asset creation, resumable upload finalization, and the updated GC batching/locking behavior.

Two follow-ups from the resumable-uploads PR, in one change.

Streaming finalize — the finalize path no longer reads the whole staged file
(up to the multi-GB cap) into memory:
- staging exposes `open(key): Blob` (a lazy `Bun.file`) instead of `readAll`.
- asset-service grows `createFromSource(blob, meta)` and an internal `ingest`
  pipeline shared with `create({ bytes })`: it stream-hashes sha256+md5 in one
  pass over the blob, sniffs via `Bun.Image(blob)` (which reads it lazily, bomb
  guard intact), then stores the blob itself. `create` wraps `ingest` with an
  in-memory Blob snapshot, so POST /assets and the seeder are unchanged.
- upload finalize streams from `staging.open(stagingKey)`; cleanup ordering,
  keep-on-transient, and the per-session lock are untouched.
- storage.store now also accepts a Blob and hands it straight to Bun.write.
  Wrapping a *file-backed* stream in a Response stalls Bun.write, so callers
  pass the Blob (not blob.stream()); the store branch was previously never
  exercised since create only ever passed a Uint8Array.

Periodic GC — expired upload sessions + their staging files were only reclaimed
opportunistically on the next upload. The API composition root now drives
`uploadService.gcExpired()` on a timer (`UPLOAD_GC_INTERVAL_MS`, default 15 min,
0 disables, capped at the 32-bit timer ceiling), error-isolated, `unref()`'d,
and cleared on shutdown.

Note: a non-race insert failure can leave a content-addressed blob unreferenced.
We intentionally don't delete it inline (a concurrent same-content upload may own
that key); orphans are benign and belong to a future reference-counted orphan-GC.

Tests: asset-service streaming-equivalence (in-memory vs source produce identical
hashes/metadata/key), gcExpired sweep/no-op, env-config GC bounds, and a real-
filesystem finalize integration test (positional chunks → file-backed streaming
hash/sniff/store → round-trip) that also guards the Bun.write(Blob) path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 33ead183-dd39-4bf9-95ec-f6c8d280353a

📥 Commits

Reviewing files that changed from the base of the PR and between f4bf42e and eece2d9.

📒 Files selected for processing (1)
  • packages/core/test/upload-service.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/core/test/upload-service.test.ts

📝 Walkthrough

Walkthrough

Adds Blob-backed staging access, a shared asset ingest path, upload finalization and expired-session GC changes, API timer wiring for periodic GC, and two standalone HTML snapshot documents.

Changes

Streaming Asset Ingest and Upload GC

Layer / File(s) Summary
Staging and storage Blob contracts
packages/storage/src/staging.ts, packages/storage/src/provider.ts, packages/storage/src/filesystem.ts
StagingStore replaces readAll with open(key): Blob; StorageProvider.store accepts Blob; filesystem storage writes Blob and Uint8Array directly through Bun.write.
AssetService shared ingest pipeline
packages/core/src/services/asset-service.ts, packages/core/test/asset-service.test.ts
Adds CreateAssetMeta, adds createFromSource to the AssetService contract, implements the shared ingest pipeline, and rewires create to delegate to it.
Upload finalize and expired-session GC
packages/db/src/repositories/upload-session-repository.ts, packages/core/src/services/upload-service.ts, packages/core/test/upload-finalize.integration.test.ts, packages/core/test/upload-service.test.ts
deleteExpired returns { token, stagingKey }[] with optional batching; upload finalization uses staging.open(...) with createFromSource; gcExpired removes staged files under the per-session mutex in batches, with updated fakes and coverage.
API GC timer and interval config
apps/api/src/env-config.ts, apps/api/src/index.ts, apps/api/test/env-config.test.ts, apps/api/test/server.test.ts
Adds MAX_TIMER_DELAY_MS and UPLOAD_GC_INTERVAL_MS; starts a periodic upload GC timer on server startup and clears it during shutdown; updates config and server tests.
Google homepage snapshot
index.html
Adds a complete HTML document for a Google homepage snapshot with inline scripts, styles, and page markup.
Facebook unsupported-browser snapshot
index.html.1
Adds a complete HTML document for a Facebook unsupported-browser snapshot with bootloader scripts, pagelet hooks, and warning content.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • deckyfx/bunbooru#2: Shares the API startup/shutdown path updated here for the GC timer wiring.
  • deckyfx/bunbooru#7: Touches the storage provider/filesystem layer that this PR extends with Blob support.
  • deckyfx/bunbooru#13: Shares the resumable upload service and session finalization path changed in this PR.

Poem

🐇 I hop through blobs with quiet glee,
No buffers stuffed in memory.
GC ticks softly, then rests at dawn,
While staged bytes stream and race is gone.
A tidy burrow, swift and bright—
This rabbit approves the new flow tonight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the two main changes: streaming upload finalize and periodic session garbage collection.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/upload-gc-and-streaming-finalize

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/api/src/env-config.ts`:
- Around line 106-113: Trim the UPLOAD_GC_INTERVAL_MS env var before parsing in
env-config.ts so whitespace-only values are treated like empty input instead of
becoming 0. Update the logic around the raw/Number(raw) handling in the interval
parser to normalize with trimming before the undefined/empty check and
validation, keeping the existing MAX_TIMER_DELAY_MS bounds check and error
message behavior.

In `@packages/core/src/services/upload-service.ts`:
- Around line 176-179: The finalize path in upload-service currently passes a
lazy staging.open(session.stagingKey) source into assetService.createFromSource
without protecting it from gcExpired(), so concurrent cleanup can delete the
staging file mid-ingest. Update the finalize flow to serialize expired-session
cleanup through the same per-session lock/recheck used by withSessionLock, or
otherwise ensure gcExpired cannot remove a staging file while createFromSource
is still reading it. Use the existing session-handling helpers in UploadService
and the createFromSource call site to keep staging cleanup and finalization
mutually safe.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 24c7c00c-a9d4-4885-b594-6e481cf04592

📥 Commits

Reviewing files that changed from the base of the PR and between a0f8a20 and ba9513b.

📒 Files selected for processing (12)
  • apps/api/src/env-config.ts
  • apps/api/src/index.ts
  • apps/api/test/env-config.test.ts
  • apps/api/test/server.test.ts
  • packages/core/src/services/asset-service.ts
  • packages/core/src/services/upload-service.ts
  • packages/core/test/asset-service.test.ts
  • packages/core/test/upload-finalize.integration.test.ts
  • packages/core/test/upload-service.test.ts
  • packages/storage/src/filesystem.ts
  • packages/storage/src/provider.ts
  • packages/storage/src/staging.ts

Comment thread apps/api/src/env-config.ts Resolved
Two findings on the streaming-finalize + GC change.

- Race (Major): gcExpired() removed staging files outside withSessionLock, but a
  finalize reads the lazy file-backed staged Blob across several passes while
  holding that lock — so a sweep of an expired session could delete the file
  mid-finalize. Serialize GC's staging removal under the same per-session lock:
  deleteExpired() now returns {token, stagingKey} so the sweep can take each
  session's lock before removing its file (waiting out any in-flight finalize,
  which on success has already removed the file → harmless no-op).
- Trim (Minor): UPLOAD_GC_INTERVAL_MS now trims before parsing, so an
  all-whitespace value takes the default instead of coercing to 0 and silently
  disabling the sweep.

Test: a concurrent gcExpired() blocks on an in-flight finalize's lock and only
removes the staging file after the finalize completes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deckyfx

deckyfx commented Jun 30, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 30, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@deckyfx

deckyfx commented Jun 30, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 30, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/db/src/repositories/upload-session-repository.ts (1)

75-79: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Consider batching expired-session deletion.

deleteExpired currently deletes and returns every expired session in one call; paired with gcExpired’s per-row staging cleanup, a large backlog can create a big DB result set and storage I/O burst. Consider adding a batch limit to this contract before the periodic timer runs in production.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/repositories/upload-session-repository.ts` around lines 75 -
79, The `deleteExpired` flow in `UploadSessionRepository` currently returns
every expired row at once, which can create a large result set and trigger too
much downstream cleanup work. Update `deleteExpired(now)` to support a batch
limit in the query/contract, and make `gcExpired` consume expired sessions in
smaller chunks instead of assuming an unbounded delete. Use the existing
`uploadSessions`, `deleteExpired`, and `gcExpired` symbols to keep the batching
behavior consistent with the periodic cleanup path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/core/test/upload-service.test.ts`:
- Around line 336-342: The gcExpired() concurrency test is currently too weak
because it checks staging.removed before confirming the expired row was actually
deleted. In upload-service.test.ts, strengthen the assertion around
service.gcExpired() by first verifying the expired row is already gone via the
same service/repository path used in the test, then assert staging.removed does
not contain "tok" so the lock-ordering check is meaningful. Keep the existing
gcExpired() and staging assertions, but make the test fail if cleanup simply has
not progressed yet.

---

Nitpick comments:
In `@packages/db/src/repositories/upload-session-repository.ts`:
- Around line 75-79: The `deleteExpired` flow in `UploadSessionRepository`
currently returns every expired row at once, which can create a large result set
and trigger too much downstream cleanup work. Update `deleteExpired(now)` to
support a batch limit in the query/contract, and make `gcExpired` consume
expired sessions in smaller chunks instead of assuming an unbounded delete. Use
the existing `uploadSessions`, `deleteExpired`, and `gcExpired` symbols to keep
the batching behavior consistent with the periodic cleanup path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 277506f4-49d9-47a5-8311-131defb75c0f

📥 Commits

Reviewing files that changed from the base of the PR and between ba9513b and 1f07653.

📒 Files selected for processing (4)
  • apps/api/src/env-config.ts
  • packages/core/src/services/upload-service.ts
  • packages/core/test/upload-service.test.ts
  • packages/db/src/repositories/upload-session-repository.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/api/src/env-config.ts

deckyfx and others added 2 commits June 30, 2026 12:43
CodeRabbit (PR #14): the concurrent-GC test asserted the staging file wasn't
removed, but that could pass simply because gcExpired() hadn't reached cleanup
yet. Also assert the expired row is already deleted, proving the sweep has
progressed past its bulk delete and is genuinely blocked on the staging lock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CodeRabbit (PR #14, nitpick): gcExpired() did one unbounded DELETE and fanned
out a Promise.all over every expired session's staging removal — a large backlog
could pull a huge result set and spawn unbounded concurrent work.

- deleteExpired(now, limit?) gains an optional cap: with a limit it deletes the
  oldest `limit` expired rows via an id subquery (Postgres DELETE has no LIMIT);
  without one it deletes all (unchanged callers).
- gcExpired loops deleteExpired(at, gcBatchSize) until a short batch, so one
  sweep drains the backlog in bounded chunks (default 500, injectable for tests).
- Drop the now-stale "completion reads the whole file into memory" rationale on
  the begin() size bound — finalize streams; the bound is a storage/abuse ceiling.

Tests: a 5-session backlog with batch size 2 drains across 3 batches (proves the
loop); verified against real Postgres that the limited subquery-DELETE returns
exactly `limit` oldest rows and leaves not-yet-expired sessions intact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deckyfx

deckyfx commented Jun 30, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 30, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/core/test/upload-service.test.ts (1)

118-118: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Don’t silently convert missing staged files into empty blobs.

open() currently returns an empty Blob when the key is absent, which can mask finalize/GC regressions as a valid zero-byte staged file. Make the fake fail on missing keys so tests preserve the staging-store contract more closely.

Proposed test-fake hardening
-      open: (key) => new Blob([Uint8Array.from(files.get(key) ?? [])]),
+      open: (key) => {
+        const file = files.get(key);
+        if (!file) throw new Error(`missing staged file: ${key}`);
+        return new Blob([Uint8Array.from(file)]);
+      },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/test/upload-service.test.ts` at line 118, The test fake for the
staging store currently turns a missing key into an empty Blob, which can hide
bugs in finalize or GC behavior. Update the open() stub in
upload-service.test.ts to fail explicitly when files.get(key) is absent instead
of returning a zero-byte Blob, so the fake matches the staging-store contract
more closely and catches regressions in the upload flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/core/test/upload-service.test.ts`:
- Line 118: The test fake for the staging store currently turns a missing key
into an empty Blob, which can hide bugs in finalize or GC behavior. Update the
open() stub in upload-service.test.ts to fail explicitly when files.get(key) is
absent instead of returning a zero-byte Blob, so the fake matches the
staging-store contract more closely and catches regressions in the upload flow.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 56427c0a-8042-469e-a3aa-6178c0f10e0d

📥 Commits

Reviewing files that changed from the base of the PR and between 1f07653 and f4bf42e.

📒 Files selected for processing (5)
  • index.html
  • index.html.1
  • packages/core/src/services/upload-service.ts
  • packages/core/test/upload-service.test.ts
  • packages/db/src/repositories/upload-session-repository.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/core/src/services/upload-service.ts

CodeRabbit (PR #14, nitpick): the test StagingStore.open() returned an empty Blob
for an absent key, which is less faithful than the real Bun.file-backed store
(reading a missing file errors) and could mask a finalize/GC that opens a stale
key. Throw on a missing key instead. All existing open() calls are preceded by
writeChunk, so behaviour is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deckyfx

deckyfx commented Jun 30, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 30, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@deckyfx
deckyfx merged commit d7bca33 into main Jun 30, 2026
2 checks passed
@deckyfx
deckyfx deleted the feat/upload-gc-and-streaming-finalize branch June 30, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant