@@ -62,6 +62,27 @@ Bug fixes
     data written by an unaffected build still read back correctly on an
     affected one — only the write path consulted the faulty check.
 
+- New ``validate-freedb-record`` hardening patch for both bundled engines:
+    ``mdb_page_alloc()`` trusted a free-database record's structure
+    completely.  Its element count was never checked against the record's
+    node size, so a forged count read past the end of the node
+    (``mdb_midl_xmerge`` SIGBUS) and merged garbage page numbers into the
+    in-memory free list.  A page number taken from a record was never
+    range-checked either — the reuse path jumps past the bound the
+    fresh-page path applies — so a crafted number could be handed out
+    directly: under ``MDB_WRITEMAP`` the store lands outside the map
+    (SIGSEGV), and without it ``mdb_page_flush()`` later pwrites at an
+    arbitrary file offset, committing silently and growing the file
+    without limit.  The record's node size must now equal
+    ``(count+1)*sizeof(pgno)``, and a reused page number must fall within
+    ``[NUM_METAS, next_pgno)`` with room for the whole run.  A forged
+    record listing the same page twice also tripped an assertion in
+    ``mdb_page_dirty()`` (``abort()``); it now returns ``MDB_CORRUPTED``.
+    As with the rest of the series this matters only if you open a database
+    file you do not trust.  These checks are structural only: a record that
+    lists a genuinely live page in range still passes, which is left to an
+    offline verifier.
+
 - New ``cve-2019-16225-validate-mp-txnid`` hardening patch for the bundled
     LMDB 1.0 engine, restoring a protection 0.9 has had since CVE-2019-16225.
     0.9 rejects a mapped page claiming ``P_DIRTY``; 1.0 removed that flag and
 
@@ -0,0 +1,133 @@
+# Detecting a forged freeDB record
+
+Status: **open**. Two confirmed defects in `mdb_page_alloc()` (Tier 0 below)
+are unfixed on both engines. They are more severe than issue #484, cheaper to
+fix, and independent of it — they should be filed separately and fixed first.
+
+Written up from an investigation into issue #484's residual freeDB caveat: the
+1.0 `mp_txnid` fix proposed there proves a page is "mine" via the allocation
+invariant, which reaches back into freeDB records that a hostile file also
+controls. The question this document answers is how such a record could be
+detected.
+
+## The short answer: detection has to start one level lower
+
+Looking for a way to detect a *semantically* forged freeDB record — one that
+lists a live page — turned up the fact that nothing validates freeDB records
+**structurally** either. No patch in either series touches `mdb_page_alloc`. A
+record read from disk is trusted completely, and this is exploitable well
+beyond the MVCC problem in #484.
+
+Two independent defects, both engines, confirmed by reproducer:
+
+**The pgno values are unbounded.** The reuse path does
+`pgno = mop[i]; goto search_done;` (`build/lib09/mdb.c:2339`,
+`build/lib10/mdb.c:2833`) — and that `goto` jumps *past* the
+`pgno + num >= env->me_maxpg` bound at `lib09:2422` / `lib10:2924`, which
+guards only the fresh-page path. So a pgno taken from the freeDB is never
+range-checked at all. Building a normal database, shifting every pgno in its
+freeDB record up by 100000 against a 10 MB map (`me_maxpg` = 2560), and
+reopening:
+
+| | result |
+| --- | --- |
+| `writemap=True` | **SIGSEGV** — the store `np->mp_pgno = pgno` lands at `me_map + 4096×100002` |
+| `writemap=False` | **completes silently**; `mdb_page_flush` pwrites at `pgno×psize`, growing `data.mdb` from 237,568 to **409,653,248 bytes** |
+
+Identical on 0.9 and 1.0. Note the non-writemap case: an attacker-controlled
+64-bit file offset, with `map_size` bypassed entirely, and *no error raised*.
+
+**The element count is unvalidated.** `i = idl[0]` (`lib09:2396`,
+`lib10:2898`) is never checked against `data.mv_size`, and
+`mdb_midl_xmerge` then reads `idl[1..idl[0]]`. Forging count 54 → 200000 on a
+440-byte record gives **SIGBUS** (reading past EOF inside the mapping) and
+merges garbage pgnos into `me_pghead`. This one is independent of the first —
+it needs no bad pgno.
+
+As a side effect, forging duplicate pgnos trips `mdb_tassert(txn, rc == 0)` in
+`mdb_page_dirty()`, which calls `abort()`. That is a denial of service where an
+`MDB_CORRUPTED` return belongs.
+
+## How to detect a forged record, in four tiers
+
+**Tier 0 — structural validation, essentially free, always-on.** Right after
+`mdb_node_read` returns the record in `mdb_page_alloc`, one linear pass over
+the IDL before it reaches `mdb_midl_xmerge`: `data.mv_size` must match
+`(idl[0]+1) * sizeof(MDB_ID)`; the entries must be strictly descending (which
+gives duplicate detection within the record for free); and every pgno must lie
+in `[NUM_METAS, txn->mt_next_pgno)`. That pass is O(n) over an array the merge
+is about to walk anyway, so it roughly doubles a cost already being paid, on a
+path that runs once per freeDB record consumed rather than per page. This
+closes both confirmed defects and needs no new flag or API.
+
+**Tier 1 — cheap semantic invariants, still always-on, partial coverage.**
+After the merge, `me_pghead` must remain strictly descending — adjacent equals
+mean two different records freed the same page without an intervening
+reallocation, which cannot happen in a sane file. On 1.0 only, a page listed in
+the record keyed by txnid K must satisfy `mp_txnid <= K`, since the page's last
+writer necessarily precedes its freeing; that catches an attacker trying to
+free a recently-written page. 0.9 cannot do this — its `MDB_page` header has no
+txnid field, only `mp_pad` and `mp_flags`. Also convert the `mdb_page_dirty`
+assertion to a `MDB_CORRUPTED` return.
+
+**Tier 2 — page accounting.** This already exists: `mdb_audit()` at
+`lib09:1831` / `lib10:2224` checks exactly
+`freecount + count + NUM_METAS == mt_next_pgno`, summing freeDB record counts
+against each DB's `md_branch_pages + md_leaf_pages + md_overflow_pages`. Adding
+a live page's pgno to a freeDB record without removing it from the tree makes
+that sum overshoot. But it is `#if (MDB_DEBUG) > 2`, it only `fprintf`s rather
+than failing, it needs all named DBs open to be correct, and — the real limit —
+it trusts `md_*_pages`, which the attacker also controls and can simply
+decrement to rebalance. It catches lazy forgeries, not deliberate ones.
+
+**Tier 3 — reachability traversal, the only sound detector.** Walk from the
+meta page through every DB root (main, freeDB, and every `F_SUBDATA` sub-DB),
+marking a bitmap of live pages, then assert that the freeDB and the live set
+are disjoint and together cover `[NUM_METAS, next_pgno)`. This trusts nothing
+but page structure, so it defeats even a fully coherent forgery. Cost is
+O(database pages) plus a bitmap of `next_pgno` bits — 32 MB for a 1 TB database
+at 4 K pages, which is fine. It cannot be a per-transaction check.
+
+## On the "no new API" constraint
+
+Tiers 0 through 2 are unconditional validation, consistent with how the rest of
+the hardening series works, and introduce no caller-visible surface. Tier 3 is
+inherently opt-in, but it does not have to be the "declare this file untrusted"
+open-time flag that #484 wanted to avoid — it fits naturally as a verb rather
+than a policy: a `verify` subcommand in `lmdb/tool.py`, or an
+`Environment.verify()` method. That is an offline audit someone runs on a file
+they do not trust, not a mode that changes how every open behaves.
+
+## What this does to issue #484
+
+The freeDB caveat on the proposed 1.0 `mp_txnid` fix survives, but shrinks a
+lot. With Tier 0 in place, a pgno arriving from the freeDB is at least
+guaranteed to be a real, in-range page, so the residual is narrowed to a
+coherent forgery naming a genuinely live page — a consistency and durability
+failure, no longer a memory-safety one. Detecting that specific case still
+requires Tier 3.
+
+## Upstream
+
+Per #484's framing, upstream does not treat a crafted `data.mdb` as an attack
+surface, so this likely stays a py-lmdb patch. An unbounded write with no error
+raised is arguably severe enough to report anyway; undecided.
+
+## Reproducers
+
+`misc/freedb_repro.py` — self-contained, engine-selectable
+(`--engine 0` / `--engine 1`), following the `misc/md_pad_repro.py` house
+style. It builds a normal database, locates its freeDB IDL, forges only the
+record bytes, and exercises each case in its own subprocess so a crash is
+isolated and its signal is visible. On current master both engines report 4 of
+5 cases not refused:
+
+| case | result |
+| --- | --- |
+| out-of-range pgno, `writemap=False` | committed silently; `data.mdb` grows 237,568 -> 409,653,248 bytes |
+| out-of-range pgno, `writemap=True` | SIGSEGV |
+| unvalidated IDL count, `writemap=False` | SIGBUS |
+| unvalidated IDL count, `writemap=True` | `MDB_CORRUPTED` (incidental downstream wrong-page-type check) |
+| duplicate pgno, `writemap=False` | SIGABRT (`mdb_page_dirty` assertion) |
+
+A minimal C reproducer and an OpenLDAP-format bug report have not been prepared.
@@ -0,0 +1,131 @@
+# Plan: offline verifier (verify-then-trust)
+
+Status: **planned**. This records a decision already reached; it introduces no
+new analysis. It is the counterpart to the runtime-hardening work tracked
+separately — see `freedb-record-validation.md` for the Tier 0 defects that
+hardening closes.
+
+## Decision: reframe the threat model
+
+No per-transaction check is 100% forge-proof short of revalidating the entire
+database on every open. Against an adversary, detection coverage is
+weakest-link: a 99% semantic detector buys nothing, because the attacker simply
+routes through the uncovered 1%. Chasing the freeDB residual — or any semantic
+forgery — inside a write transaction is therefore a losing game.
+
+So we stop chasing per-transaction semantic detection and adopt **verify-then-trust**:
+
+- Tell users, in user-facing documentation, not to open files they do not
+  control.
+- Provide an offline tool that fully verifies a file. Once a file passes, the
+  user may treat it as trusted.
+
+Everything undetectable per-transaction is trivially detectable at rest, because
+at rest there is no legitimate dirty state to be ambiguous about: a reachable
+page carrying a dirty marker, a live page listed in the freeDB, an `mp_txnid`
+above the committed meta's — each is simply invalid for a verifier with a global
+view. The online problem is hard because of ambiguity that does not exist
+offline.
+
+This retires the in-engine semantic #484 fixes previously proposed (the 0.9
+dirty-list sort and the 1.0 allocation invariant). Those are exactly the
+per-transaction semantic-detection layer this reframe replaces; the 1.0 one was
+already carrying the freeDB caveat that motivated the reframe.
+
+## Two layers, crisp identities
+
+- **Engine (runtime hardening, worked separately):** guarantees no crash, no
+  out-of-bounds access, no unbounded write — in O(1) per guard, constant-time
+  and side-effect-free, at point of use. Anything requiring iteration over
+  file-derived data does *not* belong here.
+- **`verify` (this plan):** owns every global invariant and everything that
+  requires looping over the file.
+
+The read-hardening series is the floor `verify` stands on: opening an arbitrary
+file to inspect or verify it must itself be survivable. The reframe does not
+make that series a mistake — it is the precondition for the tool.
+
+## Deliverable shape: a verb, not a policy
+
+- `python -m lmdb verify <path>` subcommand in `lmdb/tool.py`, and/or an
+  `Environment.verify()` method.
+- An offline audit someone runs on a file they do not trust — **not** an
+  open-time flag that changes how every caller behaves. This is the "untrusted
+  file" surface #484 wanted to avoid, reframed as an action rather than a mode.
+- Being Python-side, it covers `LMDB_PURE`, system-lib, and CFFI builds, which
+  the patch series never reaches.
+- Precedent: SQLite's `PRAGMA integrity_check`, BerkeleyDB's `db_verify`.
+  Verify-then-trust is the established model for file-format trust.
+
+## What "fully verify" must check (invariant catalog)
+
+"Fully verify" must mean *every invariant the write paths assume*, or the tool
+licenses trust the engine does not actually get — and the 99% problem reappears
+one level up, now stamped as approved. At minimum:
+
+- Meta-page selection matching each engine's open semantics.
+- Full reachability: main DB, every `F_SUBDATA` sub-DB, dup sub-trees, overflow
+  chains, and the freeDB itself — marking a bitmap of live pages.
+- The structural bounds the read patches already enforce (`mp_lower`/`mp_upper`,
+  LEAF2 pad, etc.).
+- Key ordering and `md_entries` / `md_*_pages` consistency. The write paths
+  assume a sorted tree; an unsorted tree that otherwise "verifies" is still
+  hostile.
+- No reachable dirty markers: on 0.9, no reachable `P_DIRTY`; on 1.0, no
+  reachable page with `mp_txnid` greater than the committed meta's txnid (this is
+  the #484 residual, trivially caught offline).
+- freeDB rules: the structural pass (strictly descending, in range, count
+  matches record size) plus the global check — the freeDB set and the live set
+  must be disjoint and together cover `[NUM_METAS, next_pgno)`.
+- Crash-consistency aware: a post-crash file legitimately contains unreachable
+  garbage and freeDB entries for pages a dead transaction touched, and must
+  still pass against the committed meta.
+
+## Operational caveats (document these, or the model is unsound)
+
+- **TOCTOU:** verify a private copy and ensure no untrusted writer can touch the
+  file afterward, or verification proves nothing.
+- **Lock file:** discard and regenerate `lock.mdb` rather than trusting one that
+  arrived with the data file.
+- **Co-writers:** "trusted" extends to every process that holds write access
+  from then on.
+
+## Design decisions to make deliberately
+
+- **Verifier trust base:** a pure-Python byte-walker cannot be owned by hostile
+  input but is slow on huge maps; an engine-backed walk is fast but spends the
+  read-hardening budget as its safety margin. Pick with eyes open.
+- **Fuzz harness:** mutate files, filter them through `verify`, then exercise the
+  engine — to validate the central claim ("passes verify ⇒ engine will not crash
+  or corrupt on it") and keep the verifier honest against upstream drift over
+  time.
+- **Cost:** O(database pages) plus a bitmap of `next_pgno` bits — ~32 MB for a
+  1 TB database at 4 K pages. Inherently not a per-transaction check.
+
+## Documentation changes
+
+- **Express the threat model in user-facing documentation for the first time.**
+  It is currently only implicit in patch notes. State it as a layered scope:
+  - Reads of arbitrary files will not crash or corrupt the process (hardened,
+    best-effort).
+  - The semantic honesty of a file is a trust property, established by provenance
+    or by `verify`.
+  - The read-hardening series is the floor this stands on.
+- Document the operational caveats above alongside the `verify` tool.
+- Update `freedb-record-validation.md`: Tier 3 becomes this plan; note Tier 0 is
+  handled in the runtime-hardening path.
+- `ChangeLog` entry when it lands.
+
+## Relationship to issue #484
+
+Close #484 by documentation plus a pointer to `verify`, and drop the in-engine
+#484 fixes. The freeDB residual that survived the online approach collapses
+offline: a live page named in a freeDB record is simply a failed
+disjoint-and-cover assertion.
+
+## Out of scope for this plan
+
+- **Tier 0 runtime hardening** — the other path, worked now (loop-free,
+  constant-time guards; see `freedb-record-validation.md`).
+- **Tier 2 page accounting** — skipped; it catches lazy forgeries, not
+  deliberate ones, and trusts `md_*_pages`, which the attacker controls.
@@ -129,6 +129,57 @@ byte-identical to pristine 0.9.36 across `mtest` through `mtest5`.
 
 ## Resolved
 
+### `validate-freedb-record` — unvalidated free-database record structure
+
+`mdb_page_alloc()` reads a free-DB record with `mdb_node_read` and trusts it
+whole. Two structural fields were never checked, on either engine:
+
+- **Element count.** `i = idl[0]` is taken as the number of page numbers in
+  the record without comparing it to the node size. A forged count drives
+  `mdb_midl_need()` and `mdb_midl_xmerge()` past the end of the node —
+  SIGBUS reading past the mapping, and garbage page numbers merged into
+  `me_pghead`.
+- **Page numbers.** The reuse path does `pgno = mop[i]; goto search_done;`,
+  jumping *past* the `pgno + num >= me_maxpg` bound that guards only the
+  fresh-page path. So a page number taken from a record is never
+  range-checked: under `MDB_WRITEMAP` the `me_map + psize*pgno` store lands
+  outside the map (SIGSEGV); without it, `mdb_page_flush()` pwrites at
+  `pgno*psize`, an attacker-controlled 64-bit file offset, committing
+  silently and growing the file without bound.
+
+A third, milder case: a record listing the same page twice makes
+`mdb_mid2l_insert` return a duplicate error, which `mdb_page_dirty()`
+funneled into `mdb_tassert` → `abort()` — a DoS where `MDB_CORRUPTED`
+belongs.
+
+The patch adds three constant-time checks, no new loop over file data:
+
+1. In `mdb_page_alloc`, before `idl[0]` is trusted, require
+   `data.mv_size` to be a non-zero multiple of `sizeof(MDB_ID)` equal to
+   `(idl[0]+1)*sizeof(MDB_ID)`. `mv_size` is read before `idl[0]` so the
+   count read itself cannot run off a short node.
+2. At `search_done`, for a reused page number (`i != 0`), require
+   `pgno` in `[NUM_METAS, mt_next_pgno)` with `num` pages of room; the
+   clause order keeps the subtraction from wrapping. `mt_next_pgno` is a
+   tighter and always-valid bound than `me_maxpg` — a freed page is by
+   definition already allocated.
+3. `mdb_page_dirty()` returns `MDB_CORRUPTED` instead of asserting, and its
+   two callers (`mdb_page_alloc`, `mdb_page_unspill`) propagate it.
+
+These are **structural** checks only (the "Tier 0" of the freeDB analysis).
+A coherent forgery that lists a genuinely live page *in range* still passes
+here and is left to an offline reachability verifier; that residual is what
+narrows, but does not close, issue #484's free-DB caveat. Because a freed
+page number now reaches the allocator guaranteed real and in-range, it can
+no longer be the memory-safety half of that caveat.
+
+Regression tests are in `tests/cve_test.py` (`FreeDbRecordTest`), engine
+agnostic so they run on both trees, and including
+`test_normal_freelist_reuse_still_works`, a positive test that passes with
+the patch removed — it guards against false rejection of a valid record.
+Each of the four forgery tests crashes (SIGBUS/SIGSEGV/SIGABRT) or commits
+silently with the patch removed, and returns `MDB_CORRUPTED` with it.
+
 ### `validate-ovpage-free` — unbounded overflow extent on the delete path
 
 `mdb_ovpage_free()` takes the extent from the page header