test_live_full_cycle still fails — 15min timeout is not a fix, platform remains unreliable

[VasilevNStas]

Problem

test_live_full_cycle consistently fails in CI because api.zerocracy.com does not complete the submitted job within the timeout window. The timeout was already increased from 2min → 5min → 15min (PR #402?), but the problem persists — the job .finished? never returns true within the deadline.

Latest example: Rultor merge run #29038320557 was cancelled after 15 minutes.

This blocks all PRs from merging since the check is required. Every contributor hits the same wall.

Analysis

The wait_for(15 * 60) approach is a band-aid, not a fix:

1. 15 minutes of wall-clock time per CI run is wasteful — it occupies a runner for a quarter of an hour doing nothing but polling
2. The problem is not the client timeout — it is that the Zerocracy platform either:
   • Does not finish the job within a reasonable time, OR
   • Does not report completion back to the API
3. No feedback loop — when the test fails, we only know "job did not finish in Ns", but not why (is the platform overloaded? is the job stuck? is the .finished? endpoint broken?)

Suggested solutions

1. Server-side (Zerocracy platform)

• Ensure jobs submitted by test_live_full_cycle complete within ~2 minutes (the actual work is trivial — a factbase insert + export)
• Or provide a dedicated CI/test endpoint that guarantees fast turnaround
• Or return job completion status synchronously for test requests

2. Client-side (this repo)

• Separate live tests from rake default — move them to a separate rake live task that runs only on schedule or on demand, not on every PR commit
• Reduce timeout back to 2–3 minutes and accept that the test may flake when the platform is under load — document it as known flaky
• Add metrics — record how long .finished? actually takes when it succeeds, to establish a real baseline

3. Process

• Allow PRs to merge even if live tests fail (they test the platform, not the client code)
• Run live tests nightly and alert on failure instead of blocking CI

Root cause

The tight coupling between client tests and the live Zerocracy platform means that any platform latency or bug directly blocks client-side development. A clear SLA or a test-specific fast path on the platform would resolve this.

[VasilevNStas]

@yegor256 please read this issue
We still have a problem.
It would be great if you could fix it.

[VasilevNStas]

@yegor256

This issue is still actively blocking all PRs from merging. Here is a detailed breakdown of why CI now takes 30+ minutes and what can be done about it.

Current state

The CI matrix was expanded from 2 jobs to 4 jobs (macOS-15 + Ubuntu-24.04 × Ruby 3.3 + 3.4). Each job runs the full suite including test_live_full_cycle.

What happened on the latest run (PR #409, run #29231732468)

Job	Started	Ended	Duration	Status
ubuntu-24.04, 3.3	07:20	—	~33min	cancelled
macos-15, 3.3	07:28	—	~25min	cancelled
macos-15, 3.4	07:37	—	~16min	cancelled
ubuntu-24.04, 3.4	—	—	—	queued (never started)

Total wall-clock time: 33 minutes, then manually cancelled.

Why it is so slow

1. test_live_full_cycle (the root cause) — this test submits a job to api.zerocracy.com and polls .finished? for up to 15 minutes. When the platform is slow or the job gets stuck, every single job in the matrix waits the full 15 minutes before the test fails. This is the single biggest contributor to CI latency.

2. Matrix expansion (amplifier) — with 4 jobs instead of 2, the probability that at least one job hits the slow platform window increases. And when a job is waiting on test_live_full_cycle, it holds a GitHub Actions runner hostage for 15 minutes doing nothing but sleep(5) + poll.

3. Runner queue time — macOS runners on GitHub Actions have limited availability. In this run, the macOS jobs started 8–17 minutes after the Ubuntu job, adding queue latency on top of the test latency.

Impact

• Every PR waits 15–35 minutes for CI to complete (or timeout)
• Developers cannot get quick feedback on their changes
• Multiple PRs are blocked waiting for green CI
• The test_live_full_cycle failure is never caused by the PR's code — it is always a platform issue

Suggested solutions

Option A: Move live tests out of rake default

Create a separate rake live task and remove test_live_* from the default suite. Live tests would still run nightly or on demand, but they would not block PR merges. This is the lowest-risk change.

Option B: Reduce timeout and mark as flaky

Reduce wait_for in test_live_full_cycle back to 2–3 minutes. The test will sometimes fail when the platform is slow, but CI will complete in 3–5 minutes instead of 15–35. A flaky test is better than a permanently slow CI.

Option C: Skip live tests in CI via environment variable

Add a guard so that test_live_* are skipped when a CI=true or TEST_LIVE=false environment variable is set. CI gets the full test suite minus the platform-dependent tests; developers can still run TEST_LIVE=true locally.

Option D: Server-side fix (Zerocracy platform)

Ensure test jobs complete within 2 minutes, or provide a dedicated fast-path endpoint for CI. This is the ideal fix but requires changes outside this repository.

I would be happy to implement any of options A, B, or C — just let me know which approach you prefer.