Repository navigation
feat: streaming upload finalize + periodic session GC - #14
Conversation
Two follow-ups from the resumable-uploads PR, in one change.
Streaming finalize — the finalize path no longer reads the whole staged file
(up to the multi-GB cap) into memory:
- staging exposes `open(key): Blob` (a lazy `Bun.file`) instead of `readAll`.
- asset-service grows `createFromSource(blob, meta)` and an internal `ingest`
pipeline shared with `create({ bytes })`: it stream-hashes sha256+md5 in one
pass over the blob, sniffs via `Bun.Image(blob)` (which reads it lazily, bomb
guard intact), then stores the blob itself. `create` wraps `ingest` with an
in-memory Blob snapshot, so POST /assets and the seeder are unchanged.
- upload finalize streams from `staging.open(stagingKey)`; cleanup ordering,
keep-on-transient, and the per-session lock are untouched.
- storage.store now also accepts a Blob and hands it straight to Bun.write.
Wrapping a *file-backed* stream in a Response stalls Bun.write, so callers
pass the Blob (not blob.stream()); the store branch was previously never
exercised since create only ever passed a Uint8Array.
Periodic GC — expired upload sessions + their staging files were only reclaimed
opportunistically on the next upload. The API composition root now drives
`uploadService.gcExpired()` on a timer (`UPLOAD_GC_INTERVAL_MS`, default 15 min,
0 disables, capped at the 32-bit timer ceiling), error-isolated, `unref()`'d,
and cleared on shutdown.
Note: a non-race insert failure can leave a content-addressed blob unreferenced.
We intentionally don't delete it inline (a concurrent same-content upload may own
that key); orphans are benign and belong to a future reference-counted orphan-GC.
Tests: asset-service streaming-equivalence (in-memory vs source produce identical
hashes/metadata/key), gcExpired sweep/no-op, env-config GC bounds, and a real-
filesystem finalize integration test (positional chunks → file-backed streaming
hash/sniff/store → round-trip) that also guards the Bun.write(Blob) path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds Blob-backed staging access, a shared asset ingest path, upload finalization and expired-session GC changes, API timer wiring for periodic GC, and two standalone HTML snapshot documents. ChangesStreaming Asset Ingest and Upload GC
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/env-config.ts`:
- Around line 106-113: Trim the UPLOAD_GC_INTERVAL_MS env var before parsing in
env-config.ts so whitespace-only values are treated like empty input instead of
becoming 0. Update the logic around the raw/Number(raw) handling in the interval
parser to normalize with trimming before the undefined/empty check and
validation, keeping the existing MAX_TIMER_DELAY_MS bounds check and error
message behavior.
In `@packages/core/src/services/upload-service.ts`:
- Around line 176-179: The finalize path in upload-service currently passes a
lazy staging.open(session.stagingKey) source into assetService.createFromSource
without protecting it from gcExpired(), so concurrent cleanup can delete the
staging file mid-ingest. Update the finalize flow to serialize expired-session
cleanup through the same per-session lock/recheck used by withSessionLock, or
otherwise ensure gcExpired cannot remove a staging file while createFromSource
is still reading it. Use the existing session-handling helpers in UploadService
and the createFromSource call site to keep staging cleanup and finalization
mutually safe.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 24c7c00c-a9d4-4885-b594-6e481cf04592
📒 Files selected for processing (12)
apps/api/src/env-config.tsapps/api/src/index.tsapps/api/test/env-config.test.tsapps/api/test/server.test.tspackages/core/src/services/asset-service.tspackages/core/src/services/upload-service.tspackages/core/test/asset-service.test.tspackages/core/test/upload-finalize.integration.test.tspackages/core/test/upload-service.test.tspackages/storage/src/filesystem.tspackages/storage/src/provider.tspackages/storage/src/staging.ts
Two findings on the streaming-finalize + GC change.
- Race (Major): gcExpired() removed staging files outside withSessionLock, but a
finalize reads the lazy file-backed staged Blob across several passes while
holding that lock — so a sweep of an expired session could delete the file
mid-finalize. Serialize GC's staging removal under the same per-session lock:
deleteExpired() now returns {token, stagingKey} so the sweep can take each
session's lock before removing its file (waiting out any in-flight finalize,
which on success has already removed the file → harmless no-op).
- Trim (Minor): UPLOAD_GC_INTERVAL_MS now trims before parsing, so an
all-whitespace value takes the default instead of coercing to 0 and silently
disabling the sweep.
Test: a concurrent gcExpired() blocks on an in-flight finalize's lock and only
removes the staging file after the finalize completes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/db/src/repositories/upload-session-repository.ts (1)
75-79: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftConsider batching expired-session deletion.
deleteExpiredcurrently deletes and returns every expired session in one call; paired withgcExpired’s per-row staging cleanup, a large backlog can create a big DB result set and storage I/O burst. Consider adding a batch limit to this contract before the periodic timer runs in production.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/repositories/upload-session-repository.ts` around lines 75 - 79, The `deleteExpired` flow in `UploadSessionRepository` currently returns every expired row at once, which can create a large result set and trigger too much downstream cleanup work. Update `deleteExpired(now)` to support a batch limit in the query/contract, and make `gcExpired` consume expired sessions in smaller chunks instead of assuming an unbounded delete. Use the existing `uploadSessions`, `deleteExpired`, and `gcExpired` symbols to keep the batching behavior consistent with the periodic cleanup path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/core/test/upload-service.test.ts`:
- Around line 336-342: The gcExpired() concurrency test is currently too weak
because it checks staging.removed before confirming the expired row was actually
deleted. In upload-service.test.ts, strengthen the assertion around
service.gcExpired() by first verifying the expired row is already gone via the
same service/repository path used in the test, then assert staging.removed does
not contain "tok" so the lock-ordering check is meaningful. Keep the existing
gcExpired() and staging assertions, but make the test fail if cleanup simply has
not progressed yet.
---
Nitpick comments:
In `@packages/db/src/repositories/upload-session-repository.ts`:
- Around line 75-79: The `deleteExpired` flow in `UploadSessionRepository`
currently returns every expired row at once, which can create a large result set
and trigger too much downstream cleanup work. Update `deleteExpired(now)` to
support a batch limit in the query/contract, and make `gcExpired` consume
expired sessions in smaller chunks instead of assuming an unbounded delete. Use
the existing `uploadSessions`, `deleteExpired`, and `gcExpired` symbols to keep
the batching behavior consistent with the periodic cleanup path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 277506f4-49d9-47a5-8311-131defb75c0f
📒 Files selected for processing (4)
apps/api/src/env-config.tspackages/core/src/services/upload-service.tspackages/core/test/upload-service.test.tspackages/db/src/repositories/upload-session-repository.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/api/src/env-config.ts
CodeRabbit (PR #14): the concurrent-GC test asserted the staging file wasn't removed, but that could pass simply because gcExpired() hadn't reached cleanup yet. Also assert the expired row is already deleted, proving the sweep has progressed past its bulk delete and is genuinely blocked on the staging lock. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CodeRabbit (PR #14, nitpick): gcExpired() did one unbounded DELETE and fanned out a Promise.all over every expired session's staging removal — a large backlog could pull a huge result set and spawn unbounded concurrent work. - deleteExpired(now, limit?) gains an optional cap: with a limit it deletes the oldest `limit` expired rows via an id subquery (Postgres DELETE has no LIMIT); without one it deletes all (unchanged callers). - gcExpired loops deleteExpired(at, gcBatchSize) until a short batch, so one sweep drains the backlog in bounded chunks (default 500, injectable for tests). - Drop the now-stale "completion reads the whole file into memory" rationale on the begin() size bound — finalize streams; the bound is a storage/abuse ceiling. Tests: a 5-session backlog with batch size 2 drains across 3 batches (proves the loop); verified against real Postgres that the limited subquery-DELETE returns exactly `limit` oldest rows and leaves not-yet-expired sessions intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/core/test/upload-service.test.ts (1)
118-118: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDon’t silently convert missing staged files into empty blobs.
open()currently returns an emptyBlobwhen the key is absent, which can mask finalize/GC regressions as a valid zero-byte staged file. Make the fake fail on missing keys so tests preserve the staging-store contract more closely.Proposed test-fake hardening
- open: (key) => new Blob([Uint8Array.from(files.get(key) ?? [])]), + open: (key) => { + const file = files.get(key); + if (!file) throw new Error(`missing staged file: ${key}`); + return new Blob([Uint8Array.from(file)]); + },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/test/upload-service.test.ts` at line 118, The test fake for the staging store currently turns a missing key into an empty Blob, which can hide bugs in finalize or GC behavior. Update the open() stub in upload-service.test.ts to fail explicitly when files.get(key) is absent instead of returning a zero-byte Blob, so the fake matches the staging-store contract more closely and catches regressions in the upload flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/core/test/upload-service.test.ts`:
- Line 118: The test fake for the staging store currently turns a missing key
into an empty Blob, which can hide bugs in finalize or GC behavior. Update the
open() stub in upload-service.test.ts to fail explicitly when files.get(key) is
absent instead of returning a zero-byte Blob, so the fake matches the
staging-store contract more closely and catches regressions in the upload flow.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 56427c0a-8042-469e-a3aa-6178c0f10e0d
📒 Files selected for processing (5)
index.htmlindex.html.1packages/core/src/services/upload-service.tspackages/core/test/upload-service.test.tspackages/db/src/repositories/upload-session-repository.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/core/src/services/upload-service.ts
CodeRabbit (PR #14, nitpick): the test StagingStore.open() returned an empty Blob for an absent key, which is less faithful than the real Bun.file-backed store (reading a missing file errors) and could mask a finalize/GC that opens a stale key. Throw on a missing key instead. All existing open() calls are preceded by writeChunk, so behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
Two follow-ups noted in the resumable-uploads PR (#13), shipped together.
Streaming finalize
The finalize path no longer reads the whole staged file (up to the multi-GB cap) into memory.
stagingexposesopen(key): Blob(a lazyBun.file) instead ofreadAll.asset-servicegrowscreateFromSource(blob, meta)+ an internalingestpipeline shared withcreate({ bytes }):Bun.Image(blob)(reads it lazily; themaxPixelsdecompression-bomb guard is intact),createwrapsingestwith an in-memoryBlobsnapshot, soPOST /assetsand the seeder are unchanged.staging.open(stagingKey); the per-session lock, cleanup ordering, and keep-on-transient behaviour are untouched.storage.storenow also accepts aBloband hands it straight toBun.write. Wrapping a file-backed stream in aResponsestallsBun.write(a real hang caught by the integration test), so callers pass the Blob, notblob.stream(). Thatstorebranch was never exercised before —createonly ever passed aUint8Array.A
Bun.Imagespike confirmed it accepts aBlob/path lazily and that streaming sha256/md5 are bit-identical to full-buffer hashing.Periodic GC
Expired upload sessions + their staging files were only reclaimed opportunistically on the next upload. The API composition root now drives
uploadService.gcExpired()on a timer:UPLOAD_GC_INTERVAL_MS(default 15 min,0disables, capped at the 32-bit timer ceiling), error-isolated,unref()'d, cleared on shutdown.Tests
gcExpiredsweep + no-op; env-config GC bounds (default/parse/0/negative/over-cap).Bun.write(Blob)path).All green:
typecheck· 154 tests ·lint:boundaries. CodeRabbit CLI pre-review applied (timer-ceiling cap); one finding (delete orphan blob on insert failure) intentionally skipped — see below.Out of scope / follow-ups
MAX_UPLOAD_BYTESnow that RAM is no longer the finalize ceiling (policy call).move()fast-path for the FS provider (can't assume FS since the store may be S3).🤖 Generated with Claude Code
Summary by CodeRabbit
UPLOAD_GC_INTERVAL_MS(defaulting to 15 minutes, with0to disable).Blobinputs.