bug: retention floor (#1065) silently deletes pre-existing execution history on upgrade — currently dev-only, ships to everyone at next release

[vybe]

Summary

#1065 (43411375, currently dev-only — not in any release; see Release exposure below) lowers five community retention defaults to 5 days. Because cleanup_service resolves each window DB-row-first with the code default as fallback, and runs a prune as a startup sweep, any pre-existing install that has no system_settings rows for those keys — which is the default state, since nothing writes them — hard-deletes its execution history within ~20 seconds of its first boot on the affected code. Silently, with /health green.

This is not a dispute of the community floor itself. That is a deliberate open-core decision and is documented in-code. The defect is narrower: a default change is applied retroactively and destructively to data that already exists, with no migration entry, no warning, and no window to react.

Observed on a managed production instance: 5352 of 5593 schedule_executions rows (95.7%) — roughly 3 months of history — deleted 20 seconds after restart. It was recoverable only because an unrelated pre-update backup happened to predate the sweep by 16 minutes. A site without that backup loses the data permanently and has no way to know it happened.

Release exposure — no released user has been affected (corrected)

This report originally said "v0.8.0". That was wrong, and the correction matters. 43411375 landed 2026-07-14, six days after the v0.8.0 tag (b3b7078b, 2026-07-08):

compare v0.8.0...43411375  ->  status: ahead, ahead_by: 646, behind_by: 0
compared against all 13 tags -> "ahead" in every case  (contained in ZERO releases)
main...43411375            ->  diverged (not on main either)

So the floor exists only on dev and feature branches. No released user has ever been exposed, and the affected instance was tracking dev, not the release. Severity is unchanged — it hits every user the moment the next release is cut — but the cohort is currently dev-trackers only, which means this is fixable before it ever reaches anyone. That is the good-news framing, and the reason to prefer the backfill fix below over a post-hoc recovery path.

Why it was misreported as v0.8.0 (a real DX trap worth its own look)

VERSION on dev reads 0.8.0 — byte-identical to the v0.8.0 tag — because it isn't bumped until the next release cuts. So GET /api/version returns 0.8.0 on an instance 646 commits past the tag, and is indistinguishable from the actual release. Anyone triaging by reported version (operators, support, this report) will mis-attribute dev-line behavior to a shipped version. Consider a dev/pre-release marker (0.8.0+dev.<sha>, or surfacing git_branch alongside version in the Build Info dialog) so the two are separable.

Component

Backend — cleanup_service + settings_service

Priority

P0 — irreversible data loss on a routine upgrade, with no operator signal.

Error

There is no error, and that is precisely the problem. The upgrade succeeds, health checks pass, and the only trace is an INFO line:

[Cleanup] Deleted 5108 schedule_executions rows older than 5 days (#772)
[Cleanup] Deleted  244 schedule_executions rows older than 5 days (#772)
[Cleanup] Startup sweep: {'execution_logs_pruned': 1223, 'execution_rows_pruned': 5108, ..., 'total': 6331}

(Two lines because each worker sweeps independently.)

Location

• src/backend/services/settings_service.py:72 — OPS_SETTINGS_DEFAULTS, the five lowered windows
• src/backend/services/settings_service.py:118 — OPS_SETTINGS_DESCRIPTIONS, which still advertises the old defaults (see factor 4)
• src/backend/services/cleanup_service.py:~140 — _read_retention_settings(), db.get_setting_value(key, OPS_SETTINGS_DEFAULTS.get(key, "0"))
• src/backend/services/cleanup_service.py — _cleanup_loop() runs run_cleanup() as a startup sweep

Root Cause

Four factors compound. Individually each is defensible; together they make silent data loss the default outcome of an upgrade.

1. A lowered default is applied retroactively to existing data. The five windows:

Key	Pre-#1065	#1065
execution_log_retention_days	30	5
execution_row_retention_days	90	5
health_check_retention_days	7	5
agent_soft_delete_retention_days	180	5
schedule_soft_delete_retention_days	30	5

An install that never touched retention has no rows, so it silently inherits the new floor and its existing history is destroyed. Changing a default for new installs is reasonable; rewriting the history of existing ones is a different act.

2. The prune runs as a startup sweep. _cleanup_loop() calls run_cleanup() immediately, so deletion happens ~20s after boot rather than at some later off-peak hour. Even a well-informed operator has no window to intervene between "backend is up on the new version" and "history is gone."

3. No docs/migrations/ entry. This is the most consequential omission. docs/migrations/ is the established convention for operator-facing changes, and tooling reads it to surface required actions before a restart. The most destructive change in v0.8.0 ships without one, so no upgrade tooling can surface it. The #1039 release note is not a substitute — it isn't in the path an upgrade actually consults.

4. The module contradicts itself. OPS_SETTINGS_DEFAULTS was updated to "5", but OPS_SETTINGS_DESCRIPTIONS still says:

"execution_row_retention_days": "Days to retain finished schedule_execution rows; rows older than this are deleted (default: 90, 0 = disabled, #772)",
"agent_soft_delete_retention_days": "Days to retain soft-deleted agents before hard-purge (default: 180, 0 = disabled, #834)",
"execution_log_retention_days": "Days to retain the JSONL transcript on schedule_executions (default: 30, 0 = disabled, #772)",

Anywhere these descriptions surface, an operator is told the default is 90 while the code deletes at 5.

5. Mass deletion logs at INFO. Destroying 95% of a table is reported at the same level as routine no-op cleanup, so it does not stand out in logs or trip any alerting.

Note that an enterprise retention entitlement does not mitigate this: enterprise/backend/retention/service.py mirrors the same floor (_OPS_DEFAULTS = {k: "5"}) and only adds a managed setter that clamps to max(value, 5). An entitled instance upgrading is exposed identically.

Reproduction Steps

1. Run any instance on code predating 43411375 for more than 5 days so schedule_executions accumulates history. Do not set any retention settings — this is the default state; nothing writes those rows.
2. Confirm none exist: GET /api/settings returns no *_retention_days keys.
3. Update to dev at or past 43411375 (or any future release containing it) and restart the backend.
4. Within ~20 seconds, every execution older than 5 days is hard-deleted. Health stays green; no error is raised.
5. SELECT MIN(started_at) FROM schedule_executions now returns a timestamp exactly 5 days old, and re-sweeps every 300s keep it there.

Suggested Fix

Preferred — backfill on upgrade, so the floor applies to new installs only. When a retention key has no row and the install predates the floor, write the previous default rather than silently applying the new one. This fully preserves the open-core intent (fresh installs get the 5-day community floor) without retroactively deleting existing users' data:

# One-off migration / startup reconciliation
_PRE_1065_DEFAULTS = {
    "execution_log_retention_days": "30",
    "execution_row_retention_days": "90",
    "health_check_retention_days": "7",
    "agent_soft_delete_retention_days": "180",
    "schedule_soft_delete_retention_days": "30",
}
# Only for installs that existed before the floor landed — a fresh DB gets the new defaults.
for key, prior in _PRE_1065_DEFAULTS.items():
    if db.get_setting_value(key, None) is None:
        db.set_setting(key, prior)

If the retroactive floor is intentional and must stand, then at minimum:

1. Ship a docs/migrations/ entry. This alone would have prevented the incident — it is the file upgrade tooling reads.
2. Do not prune destructively on the first boot after an upgrade. Skip the startup sweep when the resolved window came from a code default rather than an explicit row, and log loudly instead.
3. Log destructive prunes at WARNING, including the row count and resolved window, so mass deletion is visible.
4. Fix OPS_SETTINGS_DESCRIPTIONS to match OPS_SETTINGS_DEFAULTS (factor 4) — this is an unambiguous inconsistency regardless of the outcome here.

A dry-run/report mode (retention_dry_run=true logging what would be deleted) would also let operators see the blast radius before committing.

Environment

• Affected commit: 43411375 (feat(retention): OSS 5-day community floor + prod-compose LOG_* fix (#1039 — OSS floor) (#1065)), authored 2026-07-14
• Branch: dev — not contained in v0.8.0 (646 commits behind it) nor any other tag; not on main
• Reported version on the affected instance: 0.8.0 — misleading, see Release exposure (the dev VERSION file still reads 0.8.0)
• Backend: PostgreSQL 16 (also applies to SQLite — the resolution path is backend-agnostic)
• OS: Ubuntu 22.04 / Docker

Related

• src/backend/services/settings_service.py (OPS_SETTINGS_DEFAULTS, OPS_SETTINGS_DESCRIPTIONS)
• src/backend/services/cleanup_service.py (_read_retention_settings, _cleanup_loop)
• #1039 (OSS retention floor), feat(retention): OSS 5-day community floor + prod-compose LOG_* fix (#1039 — OSS floor) #1065 (this change), feat: implement execution log retention and pruning policy #772 (execution/health retention), Soft-delete all entities + configurable data retention policy #834 (soft-delete retention)

[vybe]

/claim

[github-actions]

Claimed by @vybe. Label status-in-progress added.

Next steps:

1. Create branch: feature/1638-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #1638

[vybe]

Correction — this report originally said v0.8.0. That was wrong. Title and body have been updated; the technical analysis is unchanged.

43411375 landed 2026-07-14, six days after the v0.8.0 tag (b3b7078b, 2026-07-08):

compare v0.8.0...43411375  ->  status: ahead, ahead_by: 646, behind_by: 0
all 13 tags ... 43411375   ->  "ahead" in every case (contained in ZERO releases)
main...43411375            ->  diverged (not on main)

No released user has ever been exposed. The floor exists only on dev and feature branches, and the affected instance was tracking dev. Severity is unchanged — it reaches every user the moment the next release is cut — but the cohort is dev-trackers today, which means this can be fixed before it ships to anyone. That strengthens the case for the backfill-on-upgrade fix over any post-hoc recovery path.

Credit for the catch goes to the reviewer who checked git tag --contains; I attributed the commit by the instance's reported version rather than by ancestry.

Which is itself the trap, and may deserve its own issue: VERSION on dev reads 0.8.0 — byte-identical to the tag — so GET /api/version returns 0.8.0 on an instance 646 commits past it. Dev-line and released builds are indistinguishable by reported version, which is exactly how the misattribution happened. A dev/pre-release marker (0.8.0+dev.<sha>) or surfacing git_branch next to version would make them separable. Happy to file that separately if useful.