Skip to content

Commit e310cb5

Browse files
committed
docs: Update P README and enforce consistent style
1 parent f282b9c commit e310cb5

12 files changed

Lines changed: 225 additions & 465 deletions

File tree

p/README.md

Lines changed: 65 additions & 124 deletions
Large diffs are not rendered by default.

p/delete-stream-anchor/README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ A model of a correct-by-construction design for reclaiming a stream's remote-tie
44

55
## The design
66

7-
Before the very first fragment is written for a stream, the replica reader writes a dedicated anchor node in Khepri, kept alive by a keep_while condition on the queue, as a blocking step whose write commits strictly before the first S3 PUT. A sweep classifies a prefix as junk when objects are present and the anchor is absent. The keep_while removes the anchor in the same transaction that deletes the queue, so the "anchor absent" signal is produced atomically and is permanent: there is no record written at deletion time that a crash could lose.
7+
Before the very first fragment is written for a stream, the replica reader writes a dedicated anchor node in Khepri, kept alive by a `keep_while` condition on the queue, as a blocking step whose write commits strictly before the first S3 PUT. A sweep classifies a prefix as junk when objects are present and the anchor is absent. The `keep_while` removes the anchor in the same transaction that deletes the queue, so the "anchor absent" signal is produced atomically and is permanent: there is no record written at deletion time that a crash could lose.
88

99
This is correct by construction off one ordering invariant. An object exists only after the anchor committed, and the anchor disappears only when the queue does, so there is no live state with objects but no anchor. Two properties fall out for free:
1010

@@ -22,14 +22,14 @@ Each is proven necessary by a gate that must fail when the requirement is droppe
2222
| Consistent reads: the sweep reads the anchor with a strongly consistent (committed) read, not a stale local replica | a stale read reports the anchor absent for a stream whose anchor just committed while S3 already shows its first fragment, and the sweep reaps live data | `tcAnchorStaleReadReapsLive` (must fail) |
2323
| Ordering: the anchor commits before the first fragment PUT | a fragment exists before the anchor commits, so a sweep in that window reaps a live stream even with a consistent read | `tcAnchorOrderingViolated` (must fail) |
2424

25-
The stale-read gate is the same shape as the `../gc-reset-multinode` stale-floor finding: a local replica cache that lags committed Khepri defeats an otherwise sound guard.
25+
The stale-read gate is the same shape as the `gc-reset-multinode` stale-floor finding: a local replica cache that lags committed Khepri defeats an otherwise sound guard.
2626

2727
## Tests
2828

2929
| Test case | Expectation |
3030
| --- | --- |
3131
| `tcAnchorReclaimsAcrossCrash` | holds: consistent reads and anchor-before-fragment reclaim the junk even when a sweep crashes mid-reap, because the absence signal is permanent |
32-
| `tcAnchorExplore` | holds across nondeterministic replication timing and a crash mid-sweep |
32+
| `tcAnchorExplore` | holds: nondeterministic replication timing and a crash mid-sweep |
3333
| `tcAnchorStaleReadReapsLive` | fails with `INV NOREAPLIVE violated: GC reaped object ... while the stream's queue is still live` |
3434
| `tcAnchorOrderingViolated` | fails with the same counterexample, from the ordering violation |
3535

@@ -45,4 +45,4 @@ p check -tc tcAnchorOrderingViolated -i 2000 # 1 bug: fragment before ancho
4545

4646
## What the model does not cover
4747

48-
The anchor's absence is a correct classifier but not a work-list. Finding the orphan prefixes still needs the deletion trigger for the prompt path (hung on the anchor node it fires for never-committed streams too) or a periodic S3-prefix scan as a backstop, which is now safe because the decision is by construction. Those provide discovery latency, not correctness.
48+
The anchor's absence is a correct classifier but not a work-list. Finding the orphan prefixes still needs the deletion trigger for the prompt path (hung on the anchor node, which it fires for never-committed streams too) or a periodic S3-prefix scan as a backstop, which is now safe because the decision is by construction. Those provide discovery latency, not correctness.

p/gc-decision/README.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,36 @@
11
# Integrated GC reap-decision
22

3-
A model of the whole rabbitmq_stream_s3_gc decision in one state space. The other GC models each prove a single guard in isolation and collapse the remaining axes into a constant: gc-reset proves still_dangling re-reads the live floor, gc-reset-multinode proves build_lookup gates on the cache epoch, gc-leading-group and gc-reset-leading-group prove the leading-group carve-out. This model composes all three classify reasons and all the guards together so they can interact, with two stores that lag independently and a sweep a reset can interleave into. The point is the interactions, not re-proving each guard.
3+
A model of the whole `rabbitmq_stream_s3_gc` decision in one state space. The other GC models each prove a single guard in isolation and collapse the remaining axes into a constant: `gc-reset` proves `still_dangling` re-reads the live floor, `gc-reset-multinode` proves `build_lookup` gates on the cache epoch, `gc-leading-group` and `gc-reset-leading-group` prove the leading-group carve-out. This model composes all three classify reasons and all the guards together so they can interact, with two stores that lag independently and a sweep a reset can interleave into. The point is the interactions, not re-proving each guard.
44

55
## The decision, as the code runs it
66

77
The sweep is three phases, each reading a different store, mirrored from the module:
88

9-
1. build_lookup reads the committed epoch with a quorum read (db:get_consistent) and reads the floor and the leading-group carve-out from the local manifest_replica cache (get_manifest_and_epoch). It proceeds only when the cache epoch equals the committed epoch, and snapshots the committed epoch together with the cached floor and carve-out.
10-
2. classify flags an object against that snapshot: a data fragment below the snapshot floor, a group below the floor that is neither the referenced leading group nor in skip-groups mode, or a manifest below the committed epoch.
11-
3. still_dangling re-validates each candidate immediately before deleting it. A stale-epoch candidate is deleted with no re-check because the epoch is monotonic. A below-floor candidate re-reads the live cache: it compares the offset to the live floor (get_manifest) and re-derives the leading-group carve-out from the live manifest.
9+
1. `build_lookup` reads the committed epoch with a quorum read (`db:get_consistent`) and reads the floor and the leading-group carve-out from the local `manifest_replica` cache (`get_manifest_and_epoch`). It proceeds only when the cache epoch equals the committed epoch, and snapshots the committed epoch together with the cached floor and carve-out.
10+
2. `classify` flags an object against that snapshot: a data fragment below the snapshot floor, a group below the floor that is neither the referenced leading group nor in skip-groups mode, or a manifest below the committed epoch.
11+
3. `still_dangling` re-validates each candidate immediately before deleting it. A stale-epoch candidate is deleted with no re-check because the epoch is monotonic. A below-floor candidate re-reads the live cache: it compares the offset to the live floor (`get_manifest`) and re-derives the leading-group carve-out from the live manifest.
1212

13-
The committed manifest truth is carried only by the authoritative referenced and unreferenced announcements, exactly as in gc-reset-multinode. The cache is the single materialized copy the sweep reads, and it may lag. A reap is a bug only when it deletes a currently referenced object.
13+
The committed manifest truth is carried only by the authoritative referenced and unreferenced announcements, exactly as in `gc-reset-multinode`. The cache is the single materialized copy the sweep reads, and it may lag. A reap is a bug only when it deletes a currently referenced object.
1414

1515
## The four guards
1616

1717
Three are shipped. Each gate turns one off and reproduces a live deletion, so the model proves it load-bearing in the composed decision rather than in isolation.
1818

1919
| Guard | Where | If removed | Gate |
2020
| --- | --- | --- | --- |
21-
| A epoch gate | build_lookup: skip unless cache epoch equals committed epoch | a node that has not applied a reset sweeps against a stale-high floor and the live re-read confirms it | `tcGcCrossNodeStaleUnguarded` |
22-
| B live-floor re-read | still_dangling: compare to the live floor, not the snapshot floor | a reset that lowers the floor between snapshot and execute lets the stale-high snapshot reap a re-tiered live fragment | `tcGcNoReread` |
23-
| C live carve-out | still_dangling: re-derive the leading group from the live manifest | a reset plus forward retention installs a new referenced leading group below the live floor that the stale snapshot carve-out deletes | `tcGcNoLeadingReread` |
21+
| A epoch gate | `build_lookup`: skip unless cache epoch equals committed epoch | a node that has not applied a reset sweeps against a stale-high floor and the live re-read confirms it | `tcGcCrossNodeStaleUnguarded` |
22+
| B live-floor re-read | `still_dangling`: compare to the live floor, not the snapshot floor | a reset that lowers the floor between snapshot and execute lets the stale-high snapshot reap a re-tiered live fragment | `tcGcNoReread` |
23+
| C live carve-out | `still_dangling`: re-derive the leading group from the live manifest | a reset plus forward retention installs a new referenced leading group below the live floor that the stale snapshot carve-out deletes | `tcGcNoLeadingReread` |
2424

2525
The fourth guard is proposed, not shipped, and is the result the model exists to surface.
2626

2727
## The gap the integration surfaces
2828

29-
With all three shipped guards on, the model still reaches a live deletion when a reset commits after the sweep has snapshotted. build_lookup samples the committed epoch once, at the start. still_dangling re-reads only the floor (get_manifest drops the epoch), so it never notices that the committed epoch has moved. On a node whose cache has not applied the reset sync, the sweep snapshots epoch N with a matching cache and an honest floor, a reset to N+1 then commits and re-tiers a live fragment below the old floor, and the live re-read returns the same stale-high floor. The epoch gate has already passed and the floor re-read defends nothing. The sweep deletes the re-tiered live fragment.
29+
With all three shipped guards on, the model still reaches a live deletion when a reset commits after the sweep has snapshotted. `build_lookup` samples the committed epoch once, at the start. `still_dangling` re-reads only the floor (`get_manifest` drops the epoch), so it never notices that the committed epoch has moved. On a node whose cache has not applied the reset sync, the sweep snapshots epoch N with a matching cache and an honest floor, a reset to N+1 then commits and re-tiers a live fragment below the old floor, and the live re-read returns the same stale-high floor. The epoch gate has already passed and the floor re-read defends nothing. The sweep deletes the re-tiered live fragment.
3030

31-
gc-reset-multinode closed the mirror image of this, where the reset commits before the snapshot so get_consistent returns the new epoch and the gate skips. Its single scenario fixed the reset-before-snapshot ordering and did not exercise reset-after-snapshot. The dangerous path is the operator CLI full sweep (run/1 to build_lookup), which pins no writer epoch and relies on the consistent read and the cache-epoch gate, both sampled once.
31+
`gc-reset-multinode` closed the mirror image of this, where the reset commits before the snapshot so `get_consistent` returns the new epoch and the gate skips. Its single scenario fixed the reset-before-snapshot ordering and did not exercise reset-after-snapshot. The dangerous path is the operator CLI full sweep (`run/1` to `build_lookup`), which pins no writer epoch and relies on the consistent read and the cache-epoch gate, both sampled once.
3232

33-
The proposed guard D re-validates freshness at the point of deletion: still_dangling re-reads the committed epoch and the cache epoch and fails closed when they differ. This requires still_dangling to use the epoch-aware read (get_manifest_and_epoch) plus a fresh get_consistent. Where that re-check lands in real code (per candidate, per list page, or once per stream before execute) is an implementation choice; the model proves only that the logical re-validation closes the window. With guard D on the model holds across the reset timing and sync delivery; with it off the shipped behavior reaps the live fragment.
33+
The proposed guard D re-validates freshness at the point of deletion: `still_dangling` re-reads the committed epoch and the cache epoch and fails closed when they differ. This requires `still_dangling` to use the epoch-aware read (`get_manifest_and_epoch`) plus a fresh `get_consistent`. Where that re-check lands in real code (per candidate, per list page, or once per stream before execute) is an implementation choice; the model proves only that the logical re-validation closes the window. With guard D on the model holds across the reset timing and sync delivery; with it off the shipped behavior reaps the live fragment.
3434

3535
## Tests
3636

@@ -64,4 +64,4 @@ p check -tc tcGcCrossNodeStaleUnguarded -i 2000 # 1 bug
6464

6565
## What the model does not cover
6666

67-
The stale-epoch reap path deletes with no re-check, and its safety rests on epoch monotonicity, which is the subject of the writer-fencing model rather than this one. The model is one stream and one sweeping node, with the rest of the cluster represented by the committed-truth announcements and the lagging cache. The fail-closed-on-quorum-error path (a partitioned node that cannot reach a quorum) is covered by gc-reset-multinode and is not repeated here.
67+
The stale-epoch reap path deletes with no re-check, and its safety rests on epoch monotonicity, which is the subject of the `writer-fencing` model rather than this one. The model is one stream and one sweeping node, with the rest of the cluster represented by the committed-truth announcements and the lagging cache. The fail-closed-on-quorum-error path (a partitioned node that cannot reach a quorum) is covered by `gc-reset-multinode` and is not repeated here.

p/gc-leading-group/README.md

Lines changed: 11 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,32 @@
11
# GC leading-group carve-out
22

3-
Models the second, independent live-deletion guard in `rabbitmq_stream_s3_gc`:
4-
`classify_group` and the `referenced_group_key` carve-out.
3+
Models the second, independent live-deletion guard in `rabbitmq_stream_s3_gc`: `classify_group` and the `referenced_group_key` carve-out.
54

6-
## The bug class
5+
## The bug
76

8-
A group object below `first_offset` is normally a dead orphan and is deleted by
9-
the offset heuristic. There is one exception: the *leading* group. On partial
10-
expiry retention advances `first_offset` *into* the leading group, so that group
11-
still sits below the floor yet remains referenced (its surviving fragments are at
12-
or above the floor). `leading_group_info/2` records that group's key as
13-
`referenced_group_key`, and `classify_group` skips it.
7+
A group object below `first_offset` is normally a dead orphan and is deleted by the offset heuristic. There is one exception, the leading group. On partial expiry retention advances `first_offset` into the leading group, so that group still sits below the floor yet remains referenced (its surviving fragments are at or above the floor). `leading_group_info/2` records that group's key as `referenced_group_key`, and `classify_group` skips it.
148

15-
Removing the carve-out (treating every group below the floor as a deletable
16-
orphan) deletes a live group - a dangling reference: a consumer reading the
17-
surviving offsets in that group can no longer fetch it.
9+
Removing the carve-out treats every group below the floor as a deletable orphan and deletes a live group, a dangling reference: a consumer reading the surviving offsets in that group can no longer fetch it.
1810

19-
This is a *distinct* guard from `still_dangling/1` (the gc-reset model): that one
20-
re-validates an offset-based finding against the live floor after a reset; this
21-
one excludes the referenced leading group from offset-based classification in the
22-
first place. The flat-offset gc-reset model does not exercise it, so it lives
23-
here.
11+
This guard is distinct from `still_dangling/1` (the `gc-reset` model): that one re-validates an offset-based finding against the live floor after a reset, this one excludes the referenced leading group from offset-based classification in the first place. The flat-offset `gc-reset` model does not exercise it, so it lives here.
2412

2513
## What the model captures
2614

27-
- Objects carry a `Kind` (`FRAGMENT` or `GROUP`); S3 holds genuine orphans below
28-
the floor, the referenced leading group (also below the floor), and a live
29-
fragment at the floor
15+
- Objects carry a `Kind` (`FRAGMENT` or `GROUP`); S3 holds genuine orphans below the floor, the referenced leading group (also below the floor), and a live fragment at the floor
3016
- The manifest records the referenced leading group key
3117
- `guardLeadingGroup` toggles the `classify_group` carve-out
3218
- `NoDanglingReference` (INV#2): GC never deletes a referenced object
3319

34-
This is a classification-correctness property, not a concurrency race, so there
35-
is no exploration test.
20+
This is a classification-correctness property, not a concurrency race, so there is no exploration test.
3621

37-
## Tests and the validation gate
22+
## Tests
3823

3924
| Test case | Expectation |
4025
| --- | --- |
41-
| `tcLeadingGroupGuarded` | **holds** - the carve-out protects the leading group; also asserts the genuine orphans are reclaimed and the leading group preserved (anti-vacuity) |
42-
| `tcLeadingGroupUnguarded` | **fails** - carve-out removed; `INV#2 violated: GC deleted referenced object (offset=80, uid=2)` |
26+
| `tcLeadingGroupGuarded` | holds: the carve-out protects the leading group, and the genuine orphans are reclaimed with the leading group preserved (anti-vacuity) |
27+
| `tcLeadingGroupUnguarded` | fails: carve-out removed; `INV#2 violated: GC deleted referenced object (offset=80, uid=2)` |
4328

44-
`tcLeadingGroupUnguarded` is *expected to fail*; that counterexample is the proof
45-
the model reproduces the live-group deletion.
29+
`tcLeadingGroupUnguarded` is expected to fail: that counterexample is the proof the model reproduces the live-group deletion.
4630

4731
```bash
4832
p compile

0 commit comments

Comments
 (0)