feat(docs): docs deployment ledger with dual-write support#16097
feat(docs): docs deployment ledger with dual-write support#16097broady wants to merge 34 commits into
Conversation
Integrate the new docs-ledger register/finish flow into the CLI's docs publish pipeline. Controlled by FERN_DOCS_DEPLOY_MODE env var: - legacy (default): existing startDocsRegister/finishDocsRegister only - dual: legacy + ledger (non-fatal if ledger fails) - ledger: ledger only (fast, incremental via CAS) The ledger path maps the resolved DocsDefinition to a DocsPublishInput (nav tree + content-addressed page blobs), calls register to get presigned URLs for missing blobs, uploads them, then calls finish. Uses raw fetch for the ledger endpoints because the oRPC contract currently types the input as DeploymentDescriptor while the server accepts DocsPublishInput (server-side structural decomposition).
…3 status handling - Use asyncPool (sliding window, concurrency=10) instead of unbounded Promise.all - Add retry with exponential backoff (1s, 2s, 4s) for transient failures (429, 5xx) - Handle S3 412 Precondition Failed as success (object already exists in CAS) - Fail fast on 403 (expired presigned URL) and 400 (content integrity mismatch) - Track and log upload stats: uploaded vs already-in-store counts - Remove stale theme field, add locale to DocsPublishInput
Collect registered API definitions during docs publishing and serialize them as a JSON blob in the ledger's apiManifest field. Previously apiManifest was always null; now it contains the full set of API definitions keyed by FDR definition ID when any are present. Updates buildLedgerInput to accept an apiDefinitions map, adds corresponding test coverage for empty and non-empty manifests, and threads the collector through publishDocs into publishDocsLedger.
…15932) * chore(cli): ignore python .venv directories * feat(cli): populate fileManifest for docs ledger publish * feat(cli): adapt to current docs-ledger and SDK contracts * fix(cli): send undefined config (DocsConfig → LedgerConfig mapping TBD) * feat(cli): map DocsConfig to LedgerConfig for docs ledger publish * test(cli): cover docs-ledger snippet payload wiring Populate the remaining snippets metadata path used during API-definition registration and add direct unit coverage for the language-specific payload builder. This keeps docs-ledger dynamic IR checks and uploads aligned with the actual snippet package/version inputs instead of empty placeholders, including the AUTO-version fallback behavior. * fix(cli): stable apiManifest serialization in buildLedgerInput Sort apiDefinitionCollector entries by key before serializing the docs-ledger apiManifest blob. Map iteration order today depends on Promise.all completion order (random run-to-run), which leaks into the manifest blob hash and therefore the docs-ledger deployment hash. Pairs with the server-side (orgId, apiName, contentHash) dedup added in fern-platform: with both fixes, byte-identical publishes produce byte-identical apiManifests and a deterministic deployment hash. * fix(cli,docs): ledger mode skips V2; stable file tokens; ADR 0009/0011/0012 wiring * fix(cli): lazy file blob loading in ledger publish to reduce memory usage Instead of holding all file Buffers in memory for the entire publish duration, store only hash→filePath mappings. Re-read files on demand during the upload step, and only for blobs the server reports as missing.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): add docs-ledger multi-locale translation publishing Port the V2 translation loop into the ledger pipeline: - buildLedgerInput now accepts an optional locale parameter (default "en") - After base deployment finishes, publishDocsViaLedger loops through resolver.getTranslationPages() and publishes each locale via register + finishTranslation - Translation MDX processing (snippet resolution, image rewrites, @/ imports, comment stripping, nav overlays) is shared with V2 - V2 registerTranslation loop is skipped when deployMode=="ledger" (translations handled by the ledger path instead) Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * fix: organize imports for biome check Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * fix: narrow docsRegistrationId type instead of non-null assertion Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * fix(cli): extract shared translation pipeline, add preview translations, parallel locale publishing Issue #1: Extract buildTranslatedDocsDefinition into shared module - Both V2 (publishDocs.ts) and ledger (publishDocsLedger.ts) now import from buildTranslatedDocsDefinition.ts instead of duplicating ~140 lines of MDX processing logic (snippet resolution, image rewrites, nav overlays) Issue #2: Wire resolver into ledger preview path - publishDocsViaLedgerPreview now accepts optional resolver parameter - When translations are available, preview publishes them via finishTranslation after the base deployment completes Issue #3: Parallel locale publishing - Replace sequential for...of with Promise.all in publishTranslationsViaLedger, matching V2's parallel approach Issue #4: Document dual-mode double-publish - Add comment explaining that dual mode intentionally publishes translations via both V2 and ledger for migration parity Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * fix(cli): reuse client in preview translations, add failure summary log 1. Reuse existing 'client' in preview translation path instead of creating a redundant client2 instance 2. Aggregate per-locale failures into a summary log at the end of translation publishing (both preview and non-preview paths): e.g. '2/5 locale(s) failed: es, ja' Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * refactor: build all locales upfront with fail-fast, single register→upload→finish flow Restructure translation publishing so all locale DocsDefinitions are built before any network calls. If any locale fails to build, the entire publish aborts immediately. Production path (publishDocsLedger.ts): - Phase 1: Build base + all translation inputs in parallel (fail-fast) - Phase 2: Merge all blobs, single register → upload → finish for base - Phase 3: Attach translations (blobs already uploaded) Preview path (publishDocsLedgerPreview.ts): - Phase 1: Build base input + all translated DocsDefinitions upfront - Phase 2: previewRegister → build translation ledger inputs (need server-assigned domain) → merge blobs → upload → finish - Phase 3: Attach translations Removes per-locale try/catch error handling — errors now propagate to abort the entire publish rather than warn-and-continue. Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> * feat(cli): pass translations inline to finish call, remove Phase 3 Replace per-locale finishTranslation loop with a single finish call that includes a translations[] array. Both production and preview paths now follow a two-phase pipeline: Phase 1: Build all locales upfront (fail-fast) Phase 2: Single register → upload → finish (base + translations) The server persists locale-specific segments for each translation entry after the base deployment is created — no separate round-trips. Co-Authored-By: Eugene Joseph <joseph.eugene@gmail.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Send the translations array in the register input (not just finish) so the server can process translation content refs and issue presigned S3 upload URLs for locale-specific blobs. Without this, translation pages with unique hashes (different from base) got no upload URL and would cause phantom S3 references after finish. Both register and finish now receive the same DocsPublishInput shape (base + translations), eliminating the asymmetry. Co-Authored-By: cbro <cbro@buildwithfern.com>
Refactor the CLI docs-ledger client to match the server's unified
locales[] wire format:
- buildLedgerInput returns { localeEntry, blobs } instead of { input, blobs }
- Deployment-level fields (orgId, domain, basepath, etc.) removed from
buildLedgerInput — assembled separately in publishDocsViaLedger
- buildAllTranslationInputs returns localeEntry per translation
- Both production and preview flows send unified locales[] array
- Base locale is locales[0], translations follow
- Tests updated to match new return shape
Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
…api/ui-core-utils to 1.1.18-e70a169b11
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } | ||
| } | ||
|
|
||
| await uploadMissingBlobs(registerResult.missingContent, blobs, context, filePaths); |
There was a problem hiding this comment.
🔴 Preview flow uploads only base-locale missing blobs, leaving translation blobs missing in CAS
In publishDocsViaLedgerPreview, the previewRegister call (line 94-109) only includes base locale data. The server returns missingContent based solely on the base locale's blobs. Translation ledger inputs are built AFTER the register call (line 118-128) and their blobs are merged into the pool (line 131-135), but uploadMissingBlobs at line 137 only uploads blobs the server said were missing from the base-locale-only register response. Translation page blobs (which have unique hashes since their markdown content differs from base pages) are never reported as missing and thus never uploaded. When client.finish is called with all locales (line 146-153), the server won't have the translation page blobs in CAS, causing the finish to fail for any preview deployment with translations.
Comparison with production flow that works correctly
In publishDocsViaLedger.ts:265-281, all locales are built upfront and included in a single publishInput sent to client.register(), so the server can identify ALL missing blobs across all locales. The preview flow should either include translation blobs in the register request, or perform a second missing-blob check after building translations.
Prompt for agents
In publishDocsViaLedgerPreview.ts, the uploadMissingBlobs call at line 137 only uses registerResult.missingContent which was computed from the base locale only (previewRegister doesn't see translation blobs). Translation page blobs are built after the register call (lines 118-128) and merged into the blobs pool (lines 131-135), but by then the server has already told us which blobs are missing — and translation-specific blobs weren't part of that response.
Fix approaches:
1. Move translation building before previewRegister, and include translation blob hashes in the register request (if the previewRegister endpoint supports it).
2. After building translations, compute which translation blob hashes aren't in the base missingContent list, and either upload them unconditionally or make a second missingContent check with the server.
3. Build the full locales array before register (matching the production flow pattern in publishDocsViaLedger) and pass all locale data to a register endpoint that accepts multiple locales.
The production flow in publishDocsViaLedger.ts works correctly because it builds all locales (line 240-256), includes them all in the register call (line 265-281), and the server returns missingContent across ALL locales.
Was this helpful? React with 👍 or 👎 to provide feedback.
…o fix ESM imports (#16099) * fix(cli): replace @fern-api/ui-core-utils with @fern-api/core-utils to fix ESM imports @fern-api/ui-core-utils@1.1.18-e70a169b11 has extensionless relative imports in its dist/index.js (e.g. "./assertNever" without .js extension). Since the package declares "type": "module", Node.js cannot resolve these at runtime. The only usage in this repo was visitDiscriminatedUnion in docs-resolver's ApiReferenceNodeConverter.ts. Replace it with the same export from the workspace package @fern-api/core-utils, which compiles correctly with .js extensions. Co-Authored-By: cbro <cbro@buildwithfern.com> * fix(cli): bump fern-typescript-sdk init fallback to 3.71.2 Cherry-pick fix from main (7e94c19). The fallback version 2.3.2 uses a Docker image with Node 20, which fails yarn install when mute-stream@4.0.0 (requiring Node 22+) is resolved as a transitive dependency. Version 3.71.2 uses Node 22 in its Docker image. Co-Authored-By: cbro <cbro@buildwithfern.com> * test: update ETE snapshots for typescript-sdk fallback version 3.71.2 Co-Authored-By: cbro <cbro@buildwithfern.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Docs Generation Benchmark ResultsComparing PR branch against median of 5 nightly run(s) on
Docs generation runs |
- Add mapIntegrations() to mapDocsConfigToLedgerConfig - Only intercom is mapped; context7 is a well-known file artifact resolved via resolveFiles (same convention as llms.txt, robots.txt) - Update comment to remove integrations from 'intentionally omitted' list - Add tests for integrations mapping in buildLedgerInput.test.ts - Add unreleased changelog entry Co-Authored-By: cbro <cbro@buildwithfern.com>
fix(cli): send correct Content-Type header for ledger file uploads uploadBlobWithRetry previously hardcoded Content-Type: application/octet-stream for all blob uploads. For file blobs (images, fonts, etc.), this caused S3 to store objects with a generic MIME type, resulting in og:image URLs serving application/octet-stream instead of image/png. Now infers MIME type from the file extension via mime.lookup() before uploading. The presigned URL enforces this content-type in its signature, so the upload header must match what FDR declared — this fix ensures they agree. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…alization (#16184) * feat: emit referencedFiles per page for ledger shard-side file materialization ADR 0014 ("File metadata on the render path"). In ledger publish, capture the file/image artifacts each page embeds and send them as `referencedFiles[]` on the page entry, so FDR can materialise the page→file join into its route shards (and drop the global files map from the read-model pointer). By the time `buildLedgerInput` assembles the payload, each page's markdown has already had image paths rewritten to `file:<sanitizedPath>` tokens (the resolver's `replaceImagePathsAndUrls` step), and in ledger mode the token suffix IS the file's `fullPath`. We scan that in-memory markdown with the SAME `file:` regex the docs bundle's MDX serializer uses, so `referencedFiles` is exactly the set the bundle resolves per page — a cheap O(content) scan, not an MDX re-parse and not a server-side re-fetch. Requires fdr-sdk carrying `PageBlobRef.referencedFiles` (fern-platform ADR-0014 PR). Stacked on feat/docs_ledger. * fix(cli): canonicalize docs ledger file references Keep resolved docs definitions path-tokenized for the ledger path, and adapt only the legacy V2 finish payload back to legacy FileIds. This prevents dual-write publishes from storing unstable file UUIDs in ledger page blobs while preserving V2 compatibility. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…on concurrency - Consolidate 4 unreleased changelog files into a single docs-ledger.yml - mapColorsV3: exhaustive switch with assertNever instead of if/else - mapMetadata: named LedgerMetadata type instead of Record<string, unknown> cast - LedgerJsFile: extract type alias outside function scope - publishDocsLedger buildAllTranslationInputs: replace unbounded Promise.all with asyncPool(TRANSLATION_BUILD_CONCURRENCY=4) - publishDocsLedgerPreview buildAllTranslationDefinitions: same asyncPool fix - publishDocsLedgerPreview previewRegisterWithLocales: named PreviewRegisterInput interface instead of Record<string, unknown> Co-Authored-By: cbro <cbro@buildwithfern.com>
…ranslated API flows) Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
…16268) fix(docs-resolver): detect changelog overview pages beyond reserved filenames Previously, ChangelogNodeConverter only recognized overview pages named 'summary', 'index', or 'overview'. Any non-date file with a different name was silently skipped, leaving overviewPageId unset and preventing frontmatter slug overrides from propagating to the V1 navigation tree. Now, when no reserved-name overview file is found, the first non-date file in the changelog directory is used as a fallback overview page. This allows its frontmatter slug override (if any) to correctly set the changelog node's slug in the V1 tree. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…#16350) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🔍 Multi-Agent Audit — Docs Deployment LedgerA 6-role review team (architect, senior eng, QA, security, release, product owner) audited this PR, with every critical/major finding independently verified to refute false positives (58 confirmed, 1 refuted). Verdict: Request changes. The ledger architecture itself is sound — content-addressed blobs (full SHA-256), stable sanitized-path file tokens, register→upload→finish with legacy-authoritative dual-write ordering, and checksum-bound presigned PUTs all verified clean. The blockers are a default-path regression, a mechanically unmergeable state, and untested orchestration. 🔴 Must Fix Before MergeMF-1 (CRITICAL) — V2 translations register sends path tokens the legacy store cannot resolve (default-mode regression)File: MF-2 (MAJOR) — Default legacy mode is not behavior-preservingFiles: MF-3 (MAJOR) — In-memory blobs are PUT with
|
| # | Item | File | Fix |
|---|---|---|---|
| S-1 | Invalid FERN_DOCS_DEPLOY_MODE silently falls back to legacy; the comment promises a warning that doesn't exist (worst failure mode for a migration flag — typo → operator believes dual is running). Flagged by 3 roles. |
docsDeployMode.ts |
Return { mode, invalidRawValue } and warn visibly, or hard-fail on unrecognized non-empty values. Pin in the existing test. |
| S-2 | uploadMissingBlobs warn-and-skips unsourceable blobs, then calls finish(). The server's HeadObject verification backstops this, but the client should fail fast with the missing hashes. |
publishDocsLedger.ts |
Throw on skip (non-fatal in dual, correctly fatal in ledger). |
| S-3 | Three mutually contradictory FileId-lifecycle comments (buildLedgerInput dedup caveat, mapDocsConfigToLedgerConfig "identity function" claim, nonexistent "finishTranslation endpoint" reference). | publishDocsLedger.ts, mapDocsConfigToLedgerConfig.ts, publishDocs.ts |
One accurate description; document the FDR content-addressed-apiDefinitionId version floor. |
| S-4 | resolveFileIdToPath never returns undefined despite its documented contract — unmapped UUIDs leak into LedgerConfig as fake paths; the mapJsFiles null guard is dead code. |
mapDocsConfigToLedgerConfig.ts |
Return undefined on miss; keep identity passthrough only for ledger mode. |
| S-5 | SVG (unmeasured) logos silently dropped from ledger config — legacy site shows logo, ledger doesn't; defeats parity checks. | mapDocsConfigToLedgerConfig.ts |
Warn when a configured logo/og-image is dropped; consider dimensionless ImageRefs. |
| S-6 | normalizeRepoUrlToHttps hardcodes github.com (wrong on GHE), doesn't strip userinfo from passthrough URLs, mangles ssh-form remotes. |
normalizeRepoUrl.ts |
Plumb GITHUB_SERVER_URL; new URL() + clear credentials in passthrough; reject git@host: forms. |
| S-7 | Wire defaultLocale is always "en" regardless of docs.yml defaultLanguage. |
publishDocsLedger.ts |
Seed base locale from docsConfig.defaultLanguage ?? "en". |
| S-8 | Preview register bypasses the typed oRPC client (raw fetch, blind cast, no retry) — contract drift won't be caught at compile time. | publishDocsLedgerPreview.ts |
Land locales[] previewRegister in the fdr-sdk contract and call through createDocsLedgerClient. |
| S-9 | Dual-write divergence is observable only as a CLI warn line — no telemetry, exit-code, or server marker. "Is the ledger ready for promotion?" is unanswerable. A ledger-only publish silently strands the legacy store readers still serve. | publishDocs.ts / FDR follow-up |
Emit a structured event on dual-write failure; have FDR record/flag ledger-only publishes. Document rollback. |
| S-10 | Changelog hygiene: description claims 4→1 consolidation but 3 ymls ship; ledger-favicon.yml describes a sub-feature of the same feature; wire-format jargon in user-facing text. Plus the undescribed monorepo-wide ui-core-utils 0.x→1.x override bump. |
changes/unreleased/, pnpm-workspace.yaml |
Fold favicon into docs-ledger.yml; trim jargon; note the ui-core-utils bump. |
| S-11 | previewId sanitized on the preview path but passed raw to publishDocsViaLedger — can split one logical preview across two server identities. |
publishDocs.ts |
Sanitize at a single choke point. |
| S-12 | Git provenance (private repo slug/branch/SHA) now durably persisted with no opt-out. | publishDocs.ts |
Document; consider honoring a telemetry opt-out; keep git fields org-gated server-side. |
🧪 Test Gaps
All five shipped test files cover only pure helpers at the happy path; everything risky lives in untested orchestration.
- Dual-write failure semantics: nothing tests that ledger failure is non-fatal in dual mode and fatal in ledger mode, that ledger mode skips
startDocsRegister, or thatcanonicalizeUploadedFilesForResolverpopulates the three id/path maps both publish paths depend on. A one-line regression in the catch block either breaks all publishes during rollout or silently drops the legacy write. - Ledger orchestration: no test that register and finish receive deep-equal inputs (the server recomputes the deployment hash on finish), that only
missingContenthashes upload, that translation blobs merge into the pool, or that preview finish forwardsdomain/basepath/previewIdverbatim. - Path sanitization & special filenames: zero coverage for spaces (demonstrably breaks
FILE_REF_PATTERNround-trip on the DEFAULT path — see MF-2), unicode NFC/NFD, case divergence, or sanitization collisions (last-write-wins manifest clobber). - Dead dedup test:
Object.values(blobs)on a Map is always[]— the "deduplicates pages with identical content" test asserts nothing. Replace withexpect([...blobs.values()].filter(b => b.toString() === markdown)).toHaveLength(1). uploadBlobWithRetrymatrix (429/5xx backoff, 412, 403/400 fail-fast, retry exhaustion) and the lazy-read MIME branch (.PNG, extensionless,.svg) — never executes in any test.- Translation pipeline:
buildTranslatedDocsDefinition(frontmatter strip, locale fallback, editThisPageUrl rewrite, page-level vs locale-level failure semantics) — pin the asymmetry before a refactor inverts it. - Round-trip property test: canonicalize →
mapDocsDefinitionToLegacyFileIdsmust be byte-identical to the pre-canonicalization legacy definition.
Note on the one refuted finding: "finish() after skipped blobs persists a corrupt deployment" is false — the FDR finish handler HeadObject-verifies every referenced blob before persisting anything and throws
PRECONDITION_FAILED. The server backstop is solid; the client-side fail-fast (S-2) is purely an ergonomics improvement.
🤖 Generated with Claude Code — multi-agent SCRUM review (29 agents, findings adversarially verified)
…ion bumps) Co-Authored-By: cbro <cbro@buildwithfern.com>
🧪 Proposed E2E / Integration TestsFollow-up to the audit above. We researched both test harnesses — the CLI's Vitest suites in this repo and FDR's There are three layers worth adding to, in priority order:
Layer 1 — CLI hermetic integration tests (highest value, lowest cost)These live alongside the PR's existing unit tests in
Two property/regression tests that pin the confirmed bugs directly: // canonicalize-roundtrip.test.ts → MF-1 / MF-2 / gap #7
// In legacy mode the canonicalize → mapDocsDefinitionToLegacyFileIds round-trip
// MUST be byte-identical to the pre-canonicalization legacy definition.
it("round-trips filenames with spaces/parens without corrupting file refs", () => {
const def = makeDocsDefinition({ files: { "image (1).png": fileId } });
const canon = canonicalizeUploadedFilesForResolver(def, /* maps */);
const back = mapDocsDefinitionToLegacyFileIds({ docsDefinition: canon, pathToFileId });
expect(back).toEqual(def); // currently FAILS — FILE_REF_PATTERN drops at "image ("
});
// translations-remap.test.ts → MF-1 (the critical default-mode regression)
it("translated definition is remapped to legacy FileIds before the V2 translations POST", () => {
// Build a translatedDefinition with a file: ref, run the publish path with a spy on the
// translations register call, assert the posted body contains UUID FileIds, not file:<path> tokens.
});The dead test from the audit (gap #4) should be fixed in the same pass — expect([...blobs.values()].filter((b) => b.toString() === markdown)).toHaveLength(1);Layer 2 — CLI E2E ledger publish (
|
Assessment of Multi-Agent Audit FindingsReviewed all 7 MF findings and the S-1 through S-12 items against the current code. Summary of actions taken and status below. MF-1 (CRITICAL) — V2 translations register sends path tokens ✅ FIXEDConfirmed. In dual mode, Fix applied: MF-2 (MAJOR) — Default legacy mode not behavior-preserving ✅ FIXEDConfirmed.
Fix applied: Both MF-3 (MAJOR) — In-memory blobs PUT with wrong Content-Type ✅ ALREADY FIXEDVerified: broady's commit The audit's note about s3mock not catching this is important — a staging test against real S3 before promoting to dual/ledger would be prudent. MF-4 (MAJOR) — Multi-locale ledger preview fails at finish ✅ ALREADY FIXEDVerified: broady's commit MF-5 (MAJOR) — Translation parity gap between V2 and ledger paths
|
| Finding | Status | Action |
|---|---|---|
| MF-1 | ✅ Fixed | mapDocsDefinitionToLegacyFileIds applied to translated definition before V2 POST |
| MF-2 | ✅ Fixed | canonicalizeUploadedFilesForResolver gated behind deployMode !== "legacy" |
| MF-3 | ✅ Already fixed | broady's Content-Type commit |
| MF-4 | ✅ Already fixed | broady's preview locales commit |
| MF-5 | Requires flow restructuring (ordering + parity) | |
| MF-6 | ℹ️ Noted | PR author decision |
| MF-7 | ℹ️ Known | Needs fdr-sdk publish |
| S-1 | ✅ Fixed | console.warn for invalid mode values |
| S-2–S-12 | ℹ️ Tracked | Follow-up items |
|
Good test proposals — the three-layer structure (hermetic → CLI E2E → FDR server) matches the actual dependency chain well. A few thoughts: Layer 1 — highest-value items agreed:
Layer 1 — note on Layer 2 — practical concern: Layer 3 — MF-3 Content-Type caveat is important: Sequencing +1 — Layer 1 now, Layer 3 with SDK publish, Layer 2 as smoke post-SDK. |
…ted definitions (MF-1, MF-2, S-1) MF-1: Apply mapDocsDefinitionToLegacyFileIds to translatedDefinition before the V2 translations register POST. Without this, dual-mode translations sent file:<sanitizedPath> tokens that the legacy store cannot resolve (it keys files by UUID). MF-2: Gate canonicalizeUploadedFilesForResolver behind deployMode !== "legacy" so pure legacy mode is byte-identical to pre-PR behavior — no fileId→sanitizedPath→UUID round-trip, no FILE_REF_PATTERN hazards, no deep-clone overhead. S-1: Emit console.warn for unrecognized FERN_DOCS_DEPLOY_MODE values instead of silently falling back to legacy. Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
…erConfig fields Remove mapIntegrations and favicon from mapDocsConfigToLedgerConfig — the current @fern-api/fdr-sdk (1.2.16-b5078a0a01) does not include these fields on LedgerConfig, so referencing them fails type-checking. These will be restored in a follow-up PR once the SDK is published with the integrations and favicon fields on LedgerConfig. Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
🔧 Requested fixes — two behavior changes that affect customers who do not opt into dual/ledgerBoth of these run regardless of (a) SDK install snippets are now registered for every
|
Reverts two behavior changes that affect all customers regardless of
deploy mode, as flagged in review:
(a) SDK snippets registration: Revert buildSnippetsConfigForSdk wiring
at the convertIrToFdrApi call site. The populated snippetsConfig
causes every SDK generate to register install snippet metadata
where it previously registered nothing. This is a desirable change
but should ship in its own PR with version-fallback guard and
changelog entry.
(b) Changelog overview fallback: Revert the implicit promotion of the
first non-date file as overview page. This could silently change
navigation for existing sites with stray non-date files. Should
ship separately with explicit opt-in (frontmatter flag) or at
minimum deterministic selection + warning.
Both changes are preserved in the branch history and can be
cherry-picked into dedicated PRs.
Co-Authored-By: cbro <cbro@buildwithfern.com>
|
Both reverted in (a) SDK snippets registrationReverted the
(b) Changelog overview fallbackReverted the Both changes are independently revertable in their own PRs now, decoupled from the ledger rollback path. Re: CI failures — the |
Co-Authored-By: cbro <cbro@buildwithfern.com>
🐞 Reproduced: MF-2 special-filename breakage still occurs in dual modeFollowed up on MF-2 with a real end-to-end publish (built CLI from this branch → local FDR + s3mock). Good news: the ReproductionFixture page referencing two images — a normal one and a spaced one: 
.png>)Legacy mode ( Dual mode ( The file was uploaded (files map has UUID Root cause
const FILE_REF_PATTERN = /file:([^\s"'<>)}\]]+)/g;The character class excludes whitespace, Scope
Suggested fixThe remap can't be a regex over arbitrary path characters. Options:
Test to addThe Verified against a real publish: CLI built from |
🧪 Dual-write: remaining test coverage needed before mergeI validated the dual-write happy path end-to-end against a local FDR ( These dual-mode behaviors are not yet covered and should be tested before merge. Listed highest-risk first. A. Dual-write resilience — ledger failure must be non-fatal (HIGHEST PRIORITY)This is the safety property the entire migration strategy rests on: in
B. MF-3 — Content-Type on the ledger CAS PUTs (needs real S3)Cannot be validated locally — adobe/s3mock does not verify the signed
C. MF-5 — byte-parity between legacy and ledger translated contentBoth stores receive the
D. API reference + SDK snippets in dual modeMy fixture had the API-reference nav stripped, so the dual-write of API definitions and the always-on
E. Custom-domain fan-out in dual modeSingle-domain only so far. The translation register fans out to
Suggested: A and C are quick local tests ( 🤖 Generated with Claude Code |
…MF-2) The FILE_REF_PATTERN regex excluded spaces, parens, quotes, and other characters from path capture, causing filenames like 'image (1).png' to be partially matched and left unremapped in dual mode. Replace the regex scan with a data-driven approach: sort known paths by length (longest first) and do exact string replacement via split/join. This handles any character that can appear in a sanitized path. Adds test case for filenames with spaces, parens, and quotes. Also updates ete-test snapshots (non-deterministic example IDs). Co-Authored-By: cbro <cbro@buildwithfern.com>
|
Fixed in // Sort paths longest-first so "docs/assets/logo.png" is replaced before "logo.png"
const sortedPaths = [...pathToFileId.keys()].sort((a, b) => b.length - a.length);
// For each known path, replace `file:<path>` occurrences via split/join
for (const path of sortedPaths) {
const fileRef = `file:${path}`;
if (!result.includes(fileRef)) continue;
const fileId = pathToFileId.get(path);
if (fileId != null) {
result = result.split(fileRef).join(`file:${fileId}`);
}
}This handles any character that can appear in a sanitized path — spaces, parens, quotes, brackets, etc. — because the replacement is data-driven rather than regex-driven. Paths are sorted longest-first to prevent prefix collisions (e.g. Added a test case covering |
|
Good audit. Quick status on each: A. Dual-write resilience (ledger failure non-fatal) — Agree this is the blocker. The catch block in B. MF-3 Content-Type on CAS PUTs — Agreed this needs real S3 and can't be validated locally. Should be gated before any environment promotion to C. MF-5 byte-parity — Good call. The translated content path for ledger uses D. API reference + snippets — Note: the E. Custom-domain fan-out — Lower risk since it's existing code that already works for legacy; the ledger path doesn't alter the domain-routing logic. But agree a multi-domain fixture is worthwhile. I'll prioritize (A) as it's the safety contract. The MF-2 fix is already pushed in |
Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
Co-Authored-By: cbro <cbro@buildwithfern.com>
3 participants
Description
Part 1 of 2 — Docs deployment ledger with dual-write support: content-addressed blob storage, multi-locale translation publishing, stable file paths for deployment-level dedup, and MIME type detection.
Part 2 (#16460, draft) adds
integrations+faviconmapping once@fern-api/fdr-sdkpublishesLedgerConfigwith those fields.Changes Made
Core ledger architecture
mapDocsConfigToLedgerConfig: Maps classicDocsConfig(FileId-based) → ledger-nativeLedgerConfig(path-based) withImageRef,PathOrUrl, typography/JS file resolutionpublishDocsLedger/publishDocsLedgerPreview: Ledger register → upload → finish flow with structured git provenancepublishDocs.ts: Dual-write integration —deployModeenv var controls"legacy"/"dual"/"ledger"paths;readAndHashFilehelper;canonicalizeUploadedFilesForResolverfor FileId ↔ sanitized path mappingdocsDeployMode.ts: Parse + validateFERN_DOCS_DEPLOY_MODEwithconsole.warnon unrecognized values (S-1)buildLedgerInput: Assemble locale entries fromDocsDefinition+ API definitionsAudit fixes (MF-1, MF-2, S-1)
translatedDefinitionremapped viamapDocsDefinitionToLegacyFileIdsbefore V2 translations POSTcanonicalizeUploadedFilesForResolvergated behinddeployMode !== "legacy"— pure legacy is byte-identical to pre-PR behaviorCode quality (review feedback)
docs-ledger.ymlLedgerMetadata,LedgerJsFile,PreviewRegisterInputmapColorsV3switch withassertNeverasyncPool(4)replacing unboundedPromise.allin translation buildingDeferred to #16460
integrationsmapping (intercom) —LedgerConfigin current SDK lacks fieldfaviconpath resolution — same SDK blockerTesting
buildLedgerInput.test.ts)Link to Devin session: https://app.devin.ai/sessions/54d7234ac66449d787042cacdf0d0ede
Requested by: @broady