Skip to content

0002-boundary-specific-serialization.md

Latest commit

 

History

History
180 lines (126 loc) · 9.48 KB

0002-boundary-specific-serialization.md

File metadata and controls

180 lines (126 loc) · 9.48 KB
Raw

ADR-0002: Use Boundary-Specific Serialization

Field Value
Status Accepted
Product Zinder
Domain Storage and serialization
Related Storage backend, Public interfaces, Service boundaries

Context

Zinder has several serialization boundaries with different failure modes:

  • Ordered RocksDB keys must preserve lexicographic order for range reads.
  • Artifact values must expose bounded metadata before decoding or decompressing large payloads.
  • Compact-block facts must serve both native WalletQuery and lightwalletd-compatible protocols without making either wire schema canonical.
  • Durable control records must survive field additions, deletions, rollback, and repair tooling.
  • Internal Rust-only binary values should optimize for low ceremony.
  • Cursor tokens must be stable, opaque to clients, small enough for gRPC metadata, and fail closed on tampering.

A single serializer either weakens ordered keys and cursors or forces Zcash protocol payloads through an unnecessary Zinder-only representation.

Decision

Zinder uses boundary-specific serialization:

  1. Ordered RocksDB keys: fixed big-endian byte layouts owned by zinder-store.
  2. Artifact values: fixed ArtifactEnvelopeHeaderV1 followed directly by a family-specific Zinder storage protobuf payload.
  3. Protocol payload bytes: protobuf bytes generated by prost from vendored lightwalletd and native Zinder .proto files only at RPC boundaries.
  4. Durable control records: storage-specific protobuf messages generated by prost, kept separate from RPC messages.
  5. Internal binary: postcard for Rust-owned, non-durable internal binary.
  6. Cursor encoding: fixed StreamCursorTokenV1 binary layout, authenticated, then base64url encoded.
  7. Materialized-view caches: no canonical format. rkyv is gated behind real workload evidence and stays private to materialized-view stores or caches.

A general serializer is forbidden for ordered store keys and cursor framing. Those bytes are protocol surfaces, not convenience structs.

Each CompactBlockArtifact stores structured block identity, time, chain metadata, and ordered wallet-scan transaction facts. zinder-store encodes that model as ZinderCompactBlockArtifactV2 behind the fixed artifact envelope. It is neither a native WalletQuery response nor a vendored lightwalletd message. Changing the storage payload shape or its semantic interpretation is a storage migration; changing an RPC schema is not automatically one.

Rationale

Fixed layouts fit boundaries where the byte layout is the contract. RocksDB keys need byte order to match logical network and height order. Cursor tokens need a stable parser, a small fixed surface, and authentication before any server-side seek. Artifact envelope headers need bounded parsing and direct payload slicing before optional decode, decompression, or checksum work.

Protobuf fits boundaries where Zinder already speaks protobuf or needs long-lived durable schema evolution. Storage protobufs preserve the consumer-neutral artifact model, while native and compatibility RPC protobufs translate it independently. Protobuf gives storage-control messages a field-number evolution model and cross-language repair tooling. Reserialized protobuf bytes are not canonical for semantic hashes, ordered keys, or cursor payloads.

postcard fits Rust-owned internal binary because it is compact, ergonomic, and stable at the wire-format level. It is not the default for durable control records because it is not self-describing and structural compatibility stays project-owned.

rkyv is valuable but only after profiling proves a derived-read workload needs it. Its zero-copy access model introduces alignment, validation, tight-buffer, format-control, and migration constraints too high-risk for the canonical store.

Storage Layout

zinder-store owns canonical key, envelope, and control-record formats. Other crates depend on domain methods, not ad-hoc byte parsing.

Fixed-layout types:

  • StoreKey: ordered RocksDB key family with explicit network and height encoding.
  • ArtifactEnvelopeHeaderV1: bounded artifact metadata parsed before payload handling.
  • StreamCursorTokenV1: fixed cursor body plus authentication tag.

Artifact envelope:

struct ArtifactEnvelopeHeaderV1 {
    magic: [u8; 4],
    envelope_version: u8,
    payload_format: PayloadFormat,
    compression_format: CompressionFormat,
    checksum_format: ChecksumFormat,
    payload_len: u32,
    uncompressed_len: u32,
}

The v1 header is fixed at 16 bytes while ChecksumFormat::None is the only supported checksum format. Future checksum support either defines a variable-length checksum tail after these fields or introduces ArtifactEnvelopeHeaderV2; it does not add permanent zero padding to every artifact row.

Cursor body:

struct StreamCursorTokenV1 {
    cursor_schema_version: u8,
    network_id: u32,
    event_sequence: u64,
    last_height: u32,
    last_hash: [u8; 32],
    flags: u8,
    auth_tag: [u8; 32],
}

The cursor body uses event_sequence (see Chain events) because retained chain-event history is keyed by event sequence. The authentication tag covers every preceding byte. The store uses HMAC-SHA256 with a per-store cursor authentication key so cursors fail closed when tampered with or replayed against another store.

Storage-specific protobuf messages follow protobuf storage discipline:

  • Storage messages are separate from RPC messages.
  • Native storage messages may reuse deleted tags only in an explicitly breaking storage-layout change that rebuilds incompatible data.
  • Vendored compatibility messages retain their upstream tag and reservation rules.
  • Required fields are avoided.
  • Enums use explicit unspecified values.
  • Migration fixtures are added before changing durable message shapes.

Names reflect domain concepts: StoreKey, ArtifactEnvelopeHeaderV1, PayloadFormat, CompressionFormat, ChecksumFormat, StreamCursorTokenV1. Names like key_codec, cursor_helper, bytes_utils, or record are forbidden.

Consequences

Positive:

  • Native and compatibility protocols share one structured semantic artifact without coupling storage to either wire schema.
  • Store keys and cursor tokens stay small, explicit, and testable with golden byte fixtures.
  • Durable control records get protobuf's mature schema-evolution model.
  • Rust-only internals keep a low-friction postcard path without making it a storage contract.
  • Future zero-copy work can target derived caches without forcing rkyv into canonical storage.

Negative:

  • Contributors learn more than one binary boundary.
  • zinder-store implements and tests fixed-layout parsers instead of deriving everything.
  • The artifact envelope and cursor token have explicit golden fixtures and fuzz/property tests.

Neutral:

  • musli and wincode remain on the watch list.
  • bitcode can remain a benchmark reference, but not a durable-storage candidate.
  • FlatBuffers and Cap'n Proto remain options only if Zinder later needs a second IDL for cross-language zero-copy data.

Switch Criteria

Replacing the fixed artifact envelope with a protobuf envelope requires both:

  • Envelope metadata evolves faster than expected.
  • Direct payload slicing is not materially useful in the production read path.

Replacing protobuf storage-control records with postcard requires all of:

  • Control records are proven Rust-only and low-change.
  • Explicit version wrappers and golden fixtures define the current reader and reject mismatched formats.
  • The switch materially reduces implementation complexity without weakening repair tooling.

Unlocking rkyv for a materialized-view store requires all of:

  • A real field-access workload improves P99 latency by at least 30%.
  • RocksDB byte access satisfies rkyv alignment and tight-buffer requirements.
  • Validation overhead is included in the benchmark.
  • Migration behavior is proven with fixtures.
  • rkyv archive types stay private to the materialized-view store or cache.

Adopting FlatBuffers or Cap'n Proto requires a new ADR.

Alternatives Considered

One Serializer Everywhere

Reduces surface area but degrades at least one boundary. Protobuf is unsuitable for ordered keys or cursor framing. postcard is not ideal for durable control records. rkyv is too risky as canonical storage. Fixed layouts cannot replace protobuf schema evolution for Zcash-compatible payloads.

Protobuf Everywhere

Strong for payloads and durable control records, but protobuf serialization is not canonical, does not provide the byte-order control needed for RocksDB keys, and adds framing ambiguity to cursor tokens.

postcard for Storage Control Records

Good Rust DX and stable wire format, but not self-describing. Durable control records need a stronger evolution and repair story than compactness alone.

rkyv for Canonical Storage

Excellent for zero-copy field access, but canonical storage needs long-term compatibility, validated reads from RocksDB values, and clear migration semantics. Those requirements are unproven for Zinder's storage shape.

FlatBuffers or Cap'n Proto

Both are credible zero-copy formats. They add a second IDL while Zinder already needs protobuf for lightwalletd compatibility. FlatBuffers Rust support remains documented as experimental. Cap'n Proto adds its own RPC model, canonicalization concepts, and traversal-limit security surface. Neither is justified.