Skip to content

#371: Extract backoff calculation into private helper#379

Merged
yegor256 merged 1 commit into
zerocracy:masterfrom
VasilevNStas:371
Jul 9, 2026
Merged

#371: Extract backoff calculation into private helper#379
yegor256 merged 1 commit into
zerocracy:masterfrom
VasilevNStas:371

Conversation

@VasilevNStas

Copy link
Copy Markdown
Contributor

Problem

The exponential backoff formula @pause * (2**attempt) was duplicated in await and recover.

Solution

Extracted into a private backoff(attempt) helper method.

Verification

  • bundle exec rubocop — 0 offenses
  • All tests pass

Closes #371.

@VasilevNStas

Copy link
Copy Markdown
Contributor Author

CI failure note: The test_live_full_cycle failure in the CI is unrelated to this PR. It's a pre-existing infrastructure issue with the Zerocracy platform (jobs not completing on the live server). See issue #354 for details and discussion.

All 118 other tests pass, and all non-live checks (rubocop, copyrights, markdown-lint, pdd, reuse, xcop, actionlint, typos, yamllint) pass successfully.

@GHX5T-SOL GHX5T-SOL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I checked current head 0d1c4f949702b705a05a01c61c07775e1c5754f6 against #371. The change extracts the duplicated @pause * (2**attempt) formula from await and recover into a private backoff(attempt) helper without changing the retry branch conditions, logging, or sleep guard.

Validation:

  • bundle check
  • ruby -c lib/baza-rb.rb
  • git diff --check upstream/master...HEAD
  • local no-network helper probe with intercepted sleep: backoff(1)=6, backoff(2)=12 for pause: 3; await retried 429 -> 200 once and recover retried 499 -> 200 once, both recording the same 6s delay as before
  • focused retry tests: 4 tests / 5 assertions
  • full non-live suites: test/test_baza_edge.rb (48 tests / 137 assertions), test/test_baza.rb (26 / 22), and test/test_fake.rb (40 / 53)
  • bundle exec rubocop: 12 files, no offenses
  • bundle exec rake picks
  • bundle exec rake yard: 96.23% documented

I did not run full local bundle exec rake because it invokes live Zerocracy API tests. Hosted Ubuntu rake on this head fails only TestBazaLive#test_live_full_cycle with Job #76630 ... did not finish in 0s; hosted macOS rake was canceled.

@0crat

0crat commented Jun 10, 2026

Copy link
Copy Markdown

@GHX5T-SOL Thanks for the review! You've earned +6 points for this contribution. Here's the breakdown per our work policy: +18 points as the baseline reward, -8 points deducted for posting absolutely no comments during review (as required by policy), and -4 points deducted for having only 12 hits-of-code (below the 24 minimum threshold). Your running score is +1432 - keep up the great work and don't forget to check your Zerocracy account! 🚀

@VasilevNStas

Copy link
Copy Markdown
Contributor Author

@yegor256 this PR is blocked by the test_live_full_cycle flaky test (#354). You mentioned in #393 (comment) that you'd fix the platform first.

All our PRs have been rebased on the test fix from #393 (increased timeout, unique job names, fixed elapsed time display). The only remaining failures are from the platform not completing jobs within 5 minutes — the code changes themselves are clean.

A review and platform fix would unblock all of these. Thanks!

@VasilevNStas

Copy link
Copy Markdown
Contributor Author

Server Status Update (29 Jun 2026)

The test_live_full_cycle CI failure is now confirmed to be a server-side infrastructure issue, not a problem with any of our PRs. Here is the evidence:

Server is alive but jobs are stuck for hours

  • GET /whoami responds in ~7s — server is reachable and authenticated
  • Balance: 8.00 Ƶ — sufficient funds
  • But jobs pushed to the server never complete within the 10-minute timeout:
Job ID Status
#79176 started 1h11m ago, taken by swarm:#4/pop/79196, 4i — only 4 iterations in over an hour
#79194 started 1h ago — no progress at all

Timeline

Time (UTC) Event
12:36 Last green master CI — test_live_full_cycle passed in 160s
12:47 First CI timeout — test_live_full_cycle failed after 622s
Now All CI runs fail, including master branch

What it means for our PRs

All our open PRs (#379, #380, #381, #382, #383, #384, #385, #387, #389, #390, #392, #393) are blocked by this same infrastructure issue. The code changes in each PR are clean and pass all unit tests.

Once the Zerocracy platform is restored (likely a restart of swarm worker #4 or clearing a stuck queue), all PRs should become green without any code changes on our side.

Suggestion: Check the swarm:#4 worker and the POP queue on the server. The fact that jobs are started but take >1 hour suggests a resource bottleneck in the compute cluster rather than a code bug.

/cc @yegor256

@VasilevNStas

Copy link
Copy Markdown
Contributor Author

Server-side root cause analysis (swarm-template)

Looking deeper at the job status we observed on the live server:

Job #79176: "started 1h11m ago, taken by swarm:#4/pop/79196, 4i"

This tells us that the job was picked up by a swarm worker (swarm:#4) and started processing. However, after over an hour it only completed 4 iterations (4i). A typical test job should converge (reach a fixed point) in seconds or a few minutes.

The connection to zerocracy/swarm-template

swarm-template is the template used to build Docker images that execute judges on the Zerocracy platform. The flow is:

  1. baza.rb calls POST /push → creates a job
  2. Server schedules it → swarm:#4 picks it up
  3. swarm:#4 runs a Docker container built from swarm-template, executing the judges
  4. Judges iterate over the factbase in cycles (Fbe.fb.query(...)) until a fixed point is reached
  5. Server reports finished? as "yes" once done

With only 4i in >1h, something is blocking the convergence.

Possible hypotheses

  1. Judge produces new facts every cycle endlessly — a judge rule keeps matching and inserting, so the fixed point is never reached. The job runs indefinitely. This could be a regression in the fbe gem or in the default judges deployed on api.zerocracy.com.

  2. Docker image cache is stale — if a new version of judges was deployed but the swarm workers still run old Docker images, they might be using outdated/buggy judge logic that loops forever.

  3. Resource contention on swarm worker rename Baza #4swarm:#4 might be overloaded with other jobs (or stuck on a previous job), causing our test job to get minimal CPU time.

  4. Factbase locking issue — if the factbase write path has a deadlock or slow I/O, each iteration takes much longer than expected.

  5. pop/79196 might itself be a stuck sub-process — the POP (proof-of-processing) may be waiting on an external service that is unresponsive.

Suggested investigation by @yegor256

Once the platform is restored, all PRs in this repo should become green — none of our code changes affect the server-side execution.

@VasilevNStas VasilevNStas force-pushed the 371 branch 2 times, most recently from e608457 to 14662b4 Compare July 6, 2026 12:46
@VasilevNStas

Copy link
Copy Markdown
Contributor Author

@yegor256 plz re-review

@yegor256 yegor256 merged commit 86c7310 into zerocracy:master Jul 9, 2026
10 checks passed
@0crat

0crat commented Jul 10, 2026

Copy link
Copy Markdown

@VasilevNStas Thanks for the contribution! You've earned +20 points for this: +24 as a basis; -4 for too few (12) hits-of-code. Please, keep them coming. Your running score is +2212; don't forget to check your Zerocracy account too).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract exponential backoff logic from await/recover into shared helper

4 participants