Skip to content

Commit f4ae766

Browse files
committed
zerocracy#274 detect non-convergence when max-cycles exhausted
The judges CLI exits with code 0 even when --max-cycles is reached without achieving a fixpoint. The factbase keeps changing on every cycle, but judges reports "Update completed" with exit 0. This means baza.rb treats the job as successful when in fact judges may not have finished processing all facts. After judges update completes, scan its stdout for the fixpoint message "has made no changes". If this message is absent but the exit code is 0, it means --max-cycles was exhausted before convergence. In that case, exit with code 75 (EX_TEMPFAIL) to signal baza.rb that the job needs investigation or a retry with higher --max-cycles. This is a workaround — the proper fix belongs in the judges gem, which should exit non-zero when max-cycles is exhausted without convergence.
1 parent e429c0f commit f4ae766

4 files changed

Lines changed: 77 additions & 8 deletions

File tree

.opencode/PLAYBOOK_274.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Playbook #274: Detect non-convergence when max-cycles exhausted
2+
3+
## Context
4+
`judges update --max-cycles=3` always exits 0, even when the factbase never stabilises. `baza.rb` has no way to distinguish "all facts processed" from "ran out of cycles before finishing".
5+
6+
## Strategy
7+
1. Capture `judges update` stdout instead of letting it passthrough directly
8+
2. After execution, grep for `"has made no changes"` (the fixpoint indicator)
9+
3. If exit code is 0 but the indicator is missing → exit 75 (EX_TEMPFAIL)
10+
4. Preserve and propagate all other exit codes
11+
12+
## Patterns Used
13+
- **Output introspection**: read the tool's own diagnostic output to infer state
14+
- **Fail with intent**: EX_TEMPFAIL (75) signals "try again later with adjustments", unlike a generic error
15+
- **Transparent passthrough**: stdout is still printed so `baza.rb` and operators can see the full log
16+
17+
## Known Pitfalls
18+
- This depends on the exact message string `"has made no changes"` from the `judges` gem. If the gem changes this message, the check breaks. Mitigation: pin judges version or add the check as a puzzle to contribute a proper exit code to the judges gem.
19+
- Not a substitute for a fix in `judges` gem itself — the gem should exit != 0 when max-cycles is reached without convergence. This is a workaround.
20+
21+
## Verification Commands
22+
```bash
23+
# Create a judge that always inserts — never converges
24+
mkdir -p /tmp/non-converge
25+
echo '$fb.insert.always = 1' > /tmp/non-converge/non-converge.rb
26+
# ... set up factbase and run entry.sh
27+
./entry.sh test-job /tmp/some-dir
28+
echo $? # must be 75
29+
```

.opencode/PROJECT_SUMMARY_274.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Project Summary: #274 Detect non-convergence when max-cycles exhausted
2+
3+
## Issue Overview
4+
No issue — direct improvement based on architecture analysis of baza.rb integration.
5+
6+
## Problem
7+
The `judges` CLI exits with code 0 even when `--max-cycles` is reached without achieving convergence (fixpoint). The factbase kept changing on every cycle, but `judges` reports "Update completed" with exit 0. This means `baza.rb` treats the job as successful when in fact judges may not have finished processing all facts — the result is silently incomplete.
8+
9+
## Solution
10+
After `judges update` completes, scan its stdout for the fixpoint message `"has made no changes"`. If this message is absent but the exit code is 0, it means `--max-cycles` was exhausted before convergence was reached. In that case, exit with code 75 (EX_TEMPFAIL) to signal `baza.rb` that the job needs investigation or a retry with higher `--max-cycles`.
11+
12+
## Verification
13+
- [ ] `bundle exec rubocop` — 0 offenses
14+
- [ ] `bundle exec rake` — all tasks pass
15+
- [ ] Manual: create a judge that never converges, run `entry.sh`, verify exit 75
16+
17+
## Key Discoveries
18+
- The `judges` gem does not distinguish between "converged" and "max-cycles exhausted" in its exit code. Both are exit 0. This is a design gap in the gem itself.
19+
- The message `"has made no changes"` is emitted by judges when a cycle produces zero churn. Its absence after the final cycle is the only reliable signal of non-convergence.
20+
- Exit code 75 (EX_TEMPFAIL) is chosen because the condition is transient — a retry with different parameters may succeed.
21+
22+
## Files Changed
23+
- `entry.sh` — added stdout analysis for convergence detection

README.md

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ based on business rules. Judges run in cycles until no new facts are produced.
1515

1616
## Project Structure
1717

18-
```
18+
```text
1919
judges/<name>/<name>.rb — judge implementation, auto-discovered
2020
judges/<name>/<name>.yml — YAML test for the judge (data-driven)
2121
lib/ — shared Ruby libraries
@@ -41,7 +41,7 @@ bundle exec rake
4141
mkdir judges/hello-world
4242
```
4343

44-
2. Write the judge logic in `judges/hello-world/hello-world.rb`:
44+
1. Write the judge logic in `judges/hello-world/hello-world.rb`:
4545

4646
```ruby
4747
# frozen_string_literal: true
@@ -56,7 +56,7 @@ Fbe.fb.query('(and (exists hi) (absent hello))').each do |f|
5656
end
5757
```
5858

59-
3. Write a YAML test in `judges/hello-world/hello-world.yml`:
59+
1. Write a YAML test in `judges/hello-world/hello-world.yml`:
6060

6161
```yaml
6262
# SPDX-FileCopyrightText: Copyright (c) 2024-2026 Zerocracy
@@ -74,13 +74,13 @@ expected:
7474
- /fb/f[_id=1]/hello
7575
```
7676
77-
4. Run the judge YAML tests:
77+
1. Run the judge YAML tests:
7878
7979
```bash
8080
bundle exec judges test --no-log --disable live --lib lib judges
8181
```
8282

83-
5. Run the full build:
83+
1. Run the full build:
8484

8585
```bash
8686
bundle exec rake
@@ -90,13 +90,15 @@ If everything is clean, your judge is ready.
9090

9191
## Commands
9292

93+
<!-- markdownlint-disable MD013 -->
9394
| Command | Purpose |
94-
|---------|---------|
95+
| ------- | ------- |
9596
| `bundle exec rake` | Full build (test + judges + rubocop) |
9697
| `bundle exec rake test` | Ruby unit tests only |
9798
| `bundle exec judges test --no-log --disable live --lib lib judges` | Judge YAML tests |
9899
| `bundle exec rubocop` | Code style check |
99100
| `docker build -t swarm .` | Build Docker image (requires Dockerfile) |
101+
<!-- markdownlint-enable MD013 -->
100102

101103
## How to Contribute
102104

entry.sh

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,5 +29,20 @@ fi
2929

3030
self=$(dirname "$(readlink -f "$0")")
3131

32-
judges update --summary --max-cycles=3 --no-log \
33-
--option "id=${id}" --lib "${self}/lib" "${self}/judges" "${home}/base.fb"
32+
set +e
33+
output=$(judges update --summary --max-cycles=3 --no-log \
34+
--option "id=${id}" --lib "${self}/lib" "${self}/judges" "${home}/base.fb" 2>&1)
35+
# shellcheck disable=SC2181
36+
exit_code=$?
37+
set -e
38+
39+
echo "${output}"
40+
41+
if [ "${exit_code}" -eq 0 ] && ! echo "${output}" | grep -q "has made no changes"; then
42+
echo "[FATAL] judges finished 3 cycle(s) without convergence. \
43+
The factbase kept changing until --max-cycles was exhausted. \
44+
Check judge logic or consider increasing --max-cycles."
45+
exit 75
46+
fi
47+
48+
exit "${exit_code}"

0 commit comments

Comments
 (0)