Skip to content

Commit 374f29b

Browse files
sandy2008claudefriedrichg
authored
fix(ingester): close TSDB on compaction failure to prevent resource leak (#7557) (#7560)
When `db.Compact()` fails after a successful `tsdb.Open()` in `createTSDB`, the opened TSDB was not closed, leaking head, WAL, and mmap file handles. Add `db.Close()` with warning-level logging on the compaction error path. Closes #7557 Signed-off-by: Sandy Chen <Yuxuan.Chen@morganstanley.com> Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
1 parent bff54c5 commit 374f29b

2 files changed

Lines changed: 4 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
* [BUGFIX] Compactor: Fix stale `cortex_bucket_index_last_successful_update_timestamp_seconds` metric not being cleaned up when tenant ownership changes due to ring rebalancing. This caused false alarms on bucket index update rate when a tenant moved between compactors. #7485
4040
* [BUGFIX] Security: Fix stored XSS vulnerability in Alertmanager and Store Gateway status pages by replacing `text/template` with `html/template`. #7512
4141
* [BUGFIX] Security: Limit decompressed gzip output in `ParseProtoReader` and OTLP ingestion path. The decompressed body is now capped by `-distributor.otlp-max-recv-msg-size`. #7515
42+
* [BUGFIX] Ingester: Close TSDB when compaction fails during `createTSDB`, preventing resource leaks (file descriptors, mmap handles) that could lead to ingester instability. #7560
4243
* [BUGFIX] Tenant Federation: Fix regex resolver clearing known users list when user scan fails. #7534
4344
* [BUGFIX] Ingester: Release the TSDB appender on every early-return path in `Push` (e.g. out-of-order label set) by deferring `Rollback`. Previously such requests leaked TSDB head series references, mmap'd chunks and pending state per request, causing the `cortex_ingester_tsdb_head_active_appenders` gauge to grow unbounded. #7528
4445
* [BUGFIX] Ring: Fix ring token conflict resolution only applied to updated instance and make constantly token conflict check during instance observe period.

pkg/ingester/ingester.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2980,6 +2980,9 @@ func (i *Ingester) createTSDB(userID string) (*userTSDB, error) {
29802980
level.Info(userLogger).Log("msg", "Running compaction after WAL replay")
29812981
err = db.Compact(context.TODO())
29822982
if err != nil {
2983+
if closeErr := db.Close(); closeErr != nil {
2984+
level.Warn(userLogger).Log("msg", "failed to close TSDB after compact failure", "err", closeErr)
2985+
}
29832986
return nil, errors.Wrapf(err, "failed to compact TSDB: %s", udir)
29842987
}
29852988

0 commit comments

Comments
 (0)