Kanban migration fails after update: no such column spawn_failures

[emm3ttarmstrong]

Bug description

After the 2026-05-06 04:00 PDT auto-update/restart, the gateway came back up but the embedded Kanban dispatcher began failing on the default board with a SQLite migration/schema error:

2026-05-06T04:02:07-07:00 armstrong-server python[4154796]: ERROR gateway.run: kanban dispatcher: tick failed on board default
Traceback (most recent call last):
  File "/home/emmett/.hermes/hermes-agent/gateway/run.py", line 3654, in _tick_once_for_board
    conn = _kb.connect(board=slug)
  File "/home/emmett/.hermes/hermes-agent/hermes_cli/kanban_db.py", line 906, in connect
    _migrate_add_optional_columns(conn)
  File "/home/emmett/.hermes/hermes-agent/hermes_cli/kanban_db.py", line 964, in _migrate_add_optional_columns
    conn.execute(
sqlite3.OperationalError: no such column: "spawn_failures"

Context

• Hermes auto-update job ran at 2026-05-06 04:00 PDT.
• hermes update --check reported the install was 190 commits behind origin/main.
• hermes update --no-backup fast-forwarded the checkout to e70e49016.
• Gateway restarted successfully at 04:01:53 and connected Telegram/Discord/Slack by 04:02:01.
• The dispatcher error appears immediately after startup when the embedded Kanban dispatcher starts.

Environment

• Repo checkout: /home/emmett/.hermes/hermes-agent
• Current commit: e70e49016 fix(cli): guard logger.debug in signal handler (#13710 regression) (#20673)
• Host: armstrong-server, Ubuntu 24.04.4 LTS
• Python: 3.11.15
• Service: hermes-gateway.service running /home/emmett/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run --replace

Expected behavior

Kanban DB migrations should be idempotent across existing board databases, and the dispatcher should not fail on startup after an update.

Actual behavior

_migrate_add_optional_columns() attempts a migration path that references spawn_failures before the column exists, causing the dispatcher tick for the default board to fail.

Acceptance criteria

• Existing Kanban board DBs from pre-update installs migrate cleanly.
• hermes kanban --board default list --json and gateway embedded dispatcher startup do not raise sqlite3.OperationalError: no such column: "spawn_failures".
• Add/adjust a migration regression test covering a DB missing spawn_failures.

[alt-glitch]

Likely regression from #20410 which renamed spawn_failures to consecutive_failures in the kanban dispatcher circuit breaker. The migration in _migrate_add_optional_columns() appears to reference spawn_failures before the column exists on DBs that predate the rename. Related: #20410.

[kshitijk4poor]

Fixed in PR #20855 (salvage of #20848 by @helix4u). The migration in _migrate_add_optional_columns now uses ADD-first-then-conditional-copy instead of RENAME COLUMN, so databases that never had spawn_failures at all (the reported crash) are handled correctly. Installs with legacy counters keep their values; installs without them get consecutive_failures = 0 as the safe default. Already live on main.

[gsurenull]

Duplicate Column Variant (same root cause)

After applying the fix for the spawn_failures migration, I hit a second variant of the same bug on a different column:

sqlite3.OperationalError: duplicate column name: consecutive_failures

Environment:

• SQLite 3.50.4
• Column consecutive_failures already exists in the tasks table (added by a previous migration)
• _migrate_add_optional_columns() still attempts ALTER TABLE ADD COLUMN consecutive_failures and fails

Root cause: The IF NOT EXISTS guard in SQLite does not reliably prevent this error when the column snapshot (cols) was taken before the column was added, but the migration was somehow interrupted or the snapshot became stale.

Local fix: Wrapped each ALTER TABLE ADD COLUMN call in _migrate_add_optional_columns() with a try/except sqlite3.OperationalError that ignores "duplicate column" errors:

try:
    conn.execute(f"ALTER TABLE tasks ADD COLUMN {col} {dtype}")
except sqlite3.OperationalError as e:
    if "duplicate column" not in str(e):
        raise
    # Column already exists, skip

Same pattern applied to all three columns: consecutive_failures, last_failure_error, and worker_pid.

---

Suggestion: The migration should either (a) use a proper schema version number instead of column-name diffing, or (b) catch and ignore duplicate column errors unconditionally since IF NOT EXISTS is unreliable in this SQLite version.

[helix4u]

This looks valid. The original #20855 fix handled the missing/legacy spawn_failures shape, but this is a different migration race: one migrator snapshots PRAGMA table_info(tasks), another migrator adds consecutive_failures, and then the first migrator still attempts the plain ALTER TABLE ... ADD COLUMN consecutive_failures.

There are two related PRs now:

• fix(kanban): serialize first-use schema initialization #21395 serializes first-use Kanban DB initialization inside one Python process. That should help the gateway notifier/dispatcher same-process race.
• [codex] Make kanban migrations tolerate duplicate columns #21021 is the more direct fix for this duplicate-column variant, because it makes _migrate_add_optional_columns() tolerate duplicate column name when another process wins the migration race.

I think #21395 is useful but not sufficient for cross-process dashboard/gateway races. We should either revive/salvage #21021 onto current main or fold its duplicate-column-tolerant migration helper into #21395 before merging.

One extra detail: for columns that have follow-up indexes, the migration should create the index whenever the column exists, not only when the current process successfully added the column. That matters if another process adds idempotency_key or task_events.run_id first and this process only observes the duplicate/existing column path.

[A-Posthuman]

Just chiming in I have the same issue:

ERROR gateway.run: kanban dispatcher: tick failed on board default
Traceback (most recent call last):
  File "/opt/hermes/gateway/run.py", line 3878, in _tick_once_for_board
    conn = _kb.connect(board=slug)
  File "/opt/hermes/hermes_cli/kanban_db.py", line 928, in connect
    _migrate_add_optional_columns(conn)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/hermes/hermes_cli/kanban_db.py", line 996, in _migrate_add_optional_columns
    conn.execute(
    ~~~~~~~~~~~~^
        "ALTER TABLE tasks ADD COLUMN consecutive_failures "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        "INTEGER NOT NULL DEFAULT 0"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
sqlite3.OperationalError: duplicate column name: consecutive_failures

[qWaitCrypto]

This looks valid. The original #20855 fix handled the missing/legacy spawn_failures shape, but this is a different migration race: one migrator snapshots PRAGMA table_info(tasks), another migrator adds consecutive_failures, and then the first migrator still attempts the plain ALTER TABLE ... ADD COLUMN consecutive_failures.

There are two related PRs now:

• fix(kanban): serialize first-use schema initialization #21395 serializes first-use Kanban DB initialization inside one Python process. That should help the gateway notifier/dispatcher same-process race.
• [codex] Make kanban migrations tolerate duplicate columns #21021 is the more direct fix for this duplicate-column variant, because it makes _migrate_add_optional_columns() tolerate duplicate column name when another process wins the migration race.

I think #21395 is useful but not sufficient for cross-process dashboard/gateway races. We should either revive/salvage #21021 onto current main or fold its duplicate-column-tolerant migration helper into #21395 before merging.

One extra detail: for columns that have follow-up indexes, the migration should create the index whenever the column exists, not only when the current process successfully added the column. That matters if another process adds idempotency_key or task_events.run_id first and this process only observes the duplicate/existing column path.

Thanks — agreed, #21395 needed the migration-layer hardening as well, not just the in-process init lock.

I’ve now updated the branch to fold in duplicate-column-tolerant optional-column migration, so it covers both:

• same-process first-use init races
• cross-process / mixed-version duplicate-column migration races

I also fixed the indexed-column detail you pointed out: CREATE INDEX IF NOT EXISTS ... now runs whenever the
migrated column exists, not only when this process added it. That now covers both tasks.idempotency_key and
task_events.run_id.

Re-ran the relevant kanban suite on the updated branch:

• python -m pytest tests/hermes_cli/test_kanban_core_functionality.py tests/hermes_cli/test_kanban_boards.py tests/ tools/test_kanban_tools.py -q
• 242 passed