Skip to content

Secondary processes need a migration-aware boot when a schema migration drops a column family #4

Description

@gustavovalverde

The schema v11 rebuild (address-index projection, docs/investigations/2026-06-10-address-index-projection.md) drops and recreates a column family at primary open. RocksDB secondaries cannot replay a drop_cf through TryCatchUpWithPrimary, so on the first post-migration boot the query secondaries error out and s6 takes the whole container down. Production required one manual railway redeploy on 2026-06-10 (~19:00Z): the second boot is clean because the primary skips the already-done migration.

The runbook (docs/runbooks/initial-sync.md, schema v11 section) documents the manual remedy. The durable fix is to make the boot survive on its own: gate secondary startup on migration completion (s6 dependency or a readiness file written by the primary after the migration commits), or have secondaries retry open with backoff while a migration marker is present.

Acceptance: a store at schema N-1 with a CF-dropping migration boots to a healthy container in one deploy, no manual restart.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions