universe: add cursor-based delta sync to federation#2202
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request optimizes the universe federation sync process by moving from a linear-diffing approach to a stateful, cursor-based delta sync model. By tracking the last sequence number processed from remote nodes, the system can now fetch only the incremental updates, drastically reducing the discovery cost for incremental syncs. The implementation includes necessary RPC updates, database schema changes for cursor persistence, and a fallback mechanism to maintain compatibility with older nodes. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements cursor-based delta sync for the universe federation, introducing the SyncDelta RPC to fetch leaves inserted after a specific sequence number and track cursors per server. This optimization significantly reduces steady-state sync traffic. Feedback on the changes suggests documenting several exported methods on countingDiffEngine to comply with the repository style guide, and refactoring the package-level syncDeltaByteBudget variable into a field on the RPCServer struct to prevent potential test brittleness.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Add FetchUniverseLeavesSince, which returns universe leaves inserted after a given sequence number (universe_leaves.id, the rowid/BIGSERIAL primary key) in insertion order, across all issuance and transfer universes. Expose it as MultiverseStore.FetchLeavesSince and via the MultiverseArchive interface and Archive, giving the RPC layer and the syncer a single access point for cursor-based delta reads. This is the read path for cursor-based federation delta sync: a peer that remembers the last sequence it saw can fetch exactly the leaves it lacks, instead of enumerating every leaf key to diff. Rewrites of existing leaves (re-org replacements) keep their sequence number and are deliberately invisible here; root comparison remains the authority on divergence.
SyncDelta returns the universe leaves inserted on a server after the given sequence number, in insertion order, each with its inclusion proof, plus the current roots of the universes the delta touches. A federation peer that tracks a per-server cursor can fetch exactly the leaves it lacks instead of enumerating every leaf key per divergent root. The handler gates each item on the universe's proof export config; filtered leaves still advance latest_seq, since the sequence is a position in the server's insertion log. Pages are additionally cut short by an approximate byte budget to stay under the default 4 MiB gRPC message limit; the first item of a page is always admitted so callers can make progress past oversized leaves. Consumption goes through the new DeltaEngine interface: the universe Archive implements it for in-process use, generating inclusion proofs on the fly, while the RpcUniverseDiff client implements it over the RPC, mapping a grpc Unimplemented status from older servers to ErrDeltaUnsupported so callers can fall back to enumeration-based sync.
Add a last_sync_seq column to universe_servers (migration 61) holding the delta sync cursor: the highest remote insertion sequence number that has been fully applied and verified locally. Expose it through the FederationLog interface as Fetch/UpsertSyncCursor. Removing and re-adding a server implicitly resets its cursor, which degrades to a bootstrap sync rather than anything incorrect.
SyncUniverseDelta pages the remote's insertion-ordered leaf delta from a cursor position, verifies each item's inclusion proof against the remote's claimed universe root, and inserts accepted items in remote insertion order — which respects proof dependencies by construction, since the remote could only have inserted a leaf after the leaves it depends on, removing any need for block-height sorting or proof-type partitioning on this path. After the delta is applied, convergence is verified per touched universe by comparing the recomputed local root against the remote root; universes that fail (tainted items, re-org rewrites, journal anomalies) are demoted to the existing enumeration-based syncRoot path. Any unresolved failure fails the run, so a successful return implies every touched universe converged and the cursor may be persisted. Sync diff events report the measured local root, not the remote's claim. The property test pins the core invariant with an in-memory MS-SMT universe harness: for any remote population and any honest cursor, delta-then-fallback converges the local side to exactly the state full enumeration sync produces, while advancing the cursor to the remote's high-water mark.
The envoy now attempts cursor-based delta sync first on each tick: fetch the per-server cursor, run SyncUniverseDelta, and persist the advanced cursor on success. Servers that don't support the delta RPC (or any delta failure at all) fall back to the existing full enumeration sync, which remains the always-correct path. Targeted syncs (SyncAssetInfo, AddServer) keep using enumeration: they are point lookups, not federation convergence. A single kill switch, --universe.no-delta-sync, forces the legacy path.
Instrument the sync fixture's remote DiffEngine with per-phase
discovery counters (round trips, enumerated items, approximate wire
bytes) and add two steady-state scenarios: all universes locally
present, one divergent by a few leaves, synced once via enumeration
and once via the delta path.
Measured at 10 universes x 400 leaves with a 4-leaf delta:
enumeration: 1 root page + 2 leaf-key pages + 4 proof fetches,
28.3 KB of enumeration for 3.9 KB of proofs (discovery_share 0.88,
and growing with universe size)
delta: 2 delta pages, zero enumeration bytes
(discovery_share 0), payload only, independent of universe size
|
@Roasbeef: review reminder |
2 participants
Previously, federation sync discovered new asset leaves by linear-diffing remote and local state (for differing asset roots) to find what was missing. This was super inefficient, in that the discovery cost scaled with the total number of leaves, rather than with the amount of new ones added.
The fix used in this PR is to adopt a stateful cursor-based model for sync. Efficiency gains are substantial, especially for incremental syncs of assets with tons of leaves, since discovery cost now scales only with the amount of data unseen. The core mechanism is as follows:
To sync, each node simply asks others "what did you add since I last checked?" with a fallback to the old enumeration sync in any pathological case (e.g. verification failure against another node's claimed root, or if the other node doesn't have a SyncDelta RPC at all). The responding node delivers entries from its log in pages, and the syncing node pages through results, advancing the cursor until it stops incrementing.
(This obviates #290, which was prototyped first, but was found to be much less efficient.)