Skip to content

#282 periodic heartbeat to prevent Docker idle timeout#282

Open
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:282-heartbeat
Open

#282 periodic heartbeat to prevent Docker idle timeout#282
VasilevNStas wants to merge 1 commit into
zerocracy:masterfrom
VasilevNStas:282-heartbeat

Conversation

@VasilevNStas

@VasilevNStas VasilevNStas commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Long-running judges update sessions can exceed Docker's stdout-idle timeout, causing the container to be killed silently.

What this PR does

Emits a timestamped [heartbeat] line to stdout every HEARTBEAT_INTERVAL seconds (default 60) while judges runs:

[heartbeat] Job 79196 is still running at 2026-06-29T20:15:30Z
[heartbeat] Job 79196 is still running at 2026-06-29T20:16:30Z

A trap on EXIT ensures the background heartbeat process is always cleaned up — even when judges fails via set -e.

Configuration

Env var Default Description
HEARTBEAT_INTERVAL 60 Seconds between heartbeat lines

Checklist

  • bundle exec rubocop — 0 offences
  • bundle exec rake — all tasks pass
  • HoC ≤ 133

@yegor256 please review

Long-running judges sessions can exceed Docker's stdout-idle
timeout, causing the container to be killed. Emit a timestamped
`[heartbeat]` line every HEARTBEAT_INTERVAL seconds (default 60)
while judges runs.

A trap on EXIT ensures the background heartbeat process is
cleaned up even when judges fails (set -e).
@VasilevNStas
VasilevNStas requested a review from yegor256 as a code owner June 29, 2026 17:26
@VasilevNStas

Copy link
Copy Markdown
Contributor Author

Why this matters

Docker kills containers that produce no stdout for a configurable idle time (commonly 5–10 minutes). A judges update that's genuinely working — e.g. processing a large factbase or waiting on external API calls — can easily exceed this threshold.

The symptom is silent: the container disappears, baza.rb sees a timeout, and nobody knows whether the job finished, crashed, or was murdered by Docker.

After this PR, a heartbeat line arrives every 60s:

[heartbeat] Job 79196 is still running at 2026-06-29T20:16:30Z

Three benefits:

  1. Docker keeps the container alive — stdout activity resets the idle timer
  2. Operators see the job making progress — the timestamp advances, proving the process isn't wedged
  3. No additional dependencies — pure bash while/sleep, no need for ts or pv

For jobs where 60s is too aggressive (or not aggressive enough), HEARTBEAT_INTERVAL is configurable per environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant