Cluster inconsistency reported after node shutdowns.

[mdre]

I have this configuration:

adb1.sh
adb2.sh
adb3.sh
adb4.sh

So Nodo1 is always the leader, then Nodo2 and Nodo3. Nodo4 is only replica.

Start the cluster.

Now kill the leader.

Nodo2 warn that Nodo1 is missing.

Now start again Nodo1. Wait until it become the leader and drop down Nodo4. Nodo1 start to claim for Nodo4. Ok.

but when Nodo4 start again, Nodo1 report an inconsistency and never stop.

[lvca]

I wrote a new test that mirrors your scenario in-process:

1. 4 nodes with priorities 10/8/6/0, last node replica role, quorumTimeout=1000ms, persistStorage=false.
2. Write 500 records → kill priority-leader → write more → restart leader (storage wiped) → wait for it to reclaim leadership via priority → write more → stop replica → write more → restart replica (storage wiped) → final writes.
3. Assert: cluster converges, all 4 nodes hold all 1700 records, page-level identical.

The test passes consistently across multiple runs. The cluster does see transient INCONSISTENCY reply and Follower failed (request=null, errorCount=N) warnings during the rejoin window (matching the screenshots), but recovers within ~1 second and reaches consistency.

I cannot reproduce the "never stops" behavior with an in-process 4-node cluster. The pattern in the screenshot (UNAVAILABLE: connection timed out after 30000 ms, ConnectTimeoutException) is consistent with gRPC channel staleness against a Docker container that changed IP on restart, not a bug in the Raft state machine or replication path:

• Java's default networkaddress.cache.ttl is 30 seconds (or -1/forever in some JDK distros). If the JVM caches DNS forever, the leader's Ratis GrpcLogAppender channel keeps trying the old IP after the container's IP changes, and it never recovers.
• Apache Ratis does not expose an API to refresh LogAppender channels mid-run; only leader change triggers refreshRaftClient (already wired in RaftHAServer.notifyLeaderChanged).

@mdre I have some questions for you. Can you please check in the user's Docker environment:

1. Output of java -XshowSettings:properties 2>&1 | grep cache.ttl inside the container - is networkaddress.cache.ttl set to -1?
2. Does nslookup arcadedb-nodo4-test from inside the leader container return the correct (new) IP after Nodo4 restarts, while the leader is still logging the loop?
3. If the answer to (1) is "forever" - adding -Dnetworkaddress.cache.ttl=10 to JAVA_OPTS is the likely fix (no code change needed).

[mdre]

There's no docker here. All are Debian 13 vm running in Proxmox environment with their own memory, disk and IP address on the same cluster.

[mdre]

I'am sending batch of 50000 records to insert and Vertex have 3 o 4 millons records. Maybe it happend there.

[mdre]

Now it is throwing this:

It stopped creating edges. So I need to stop the migration test.

[mdre]

I see some data lost on some vertex but not sure why. I will enhance the test.

[lvca]

Thanks for the build hash — 95c57ba9b already contains the fixes for #4063 (compaction-during-replication causing missing index files on followers) and #4076 (data loss on massive insert with WAL disabled). So you're not hitting those bugs.

The symptoms you're now showing — Cannot find indexes [...] Ignoring them warnings on multiple types, edges no longer being created, and observed vertex data loss — overlap heavily with the signatures of those two bugs but not their root causes (since they're patched). That tells me one of three things is happening:

1. Pre-existing schema corruption in your database files. The c374693 fix prevents new index/file divergence; it does not repair followers (or a former-leader-now-replica) that already carried a broken schema-to-file mapping from a previous run on an older build. If those VMs were ever running an earlier version when this migration was first attempted, their /var/arcadeDatabases may still hold an inconsistent schema JSON ↔ file set. The "Cannot find indexes" warnings are coming from LocalSchema.load() reading the on-disk schema JSON — that state predates startup and is not produced by the current run.

2. A different code path with the same surface. With 50k-record batches against 3-4M-vertex types, GraphBatch's edge-only flush path is hammered and bypasses many of the assumptions used in the regression tests for Index warnings from cluster FOLLOWER #4063 / The cluster didn't replicate data on massive insert with gRPC. #4076. There may be a remaining race that those fixes did not cover.

3. The original issue (this ticket) is downstream of one of the above. The leader's "INCONSISTENCY never stops" loop you reported first looks consistent with a follower whose state-machine apply keeps failing because it can't find an index file referenced by an incoming schema entry — applySchemaEntry then keeps failing, the entry never advances, and the leader sees the same nextIndex repeatedly.

To unblock this, can you do two things:

A. Clean-slate test. On all 4 VMs, stop the server, delete the database directory contents (/var/arcadeDatabases/<dbName>/*) and the Raft storage (raft-storage-*), bring the cluster up empty on 95c57ba9b, and re-run the migration from scratch. If that completes cleanly, the issue was carried-over corruption from a pre-fix run. If it still fails, it's a remaining bug and we need (B).

B. Capture during the failure. Whichever node is logging the loop, run before stopping it:

• curl -u root:<pwd> http://<that-node>:2480/api/v1/cluster — full cluster status (term, commit index, lag, matchIndex/nextIndex per peer)
• The complete logs/ directory from each of the 4 nodes for the failing window
• The output of ls -la /var/arcadeDatabases/<dbName>/ from each node, plus the schema JSON (grep -c indexes /var/arcadeDatabases/<dbName>/schema.json)

One more variable I'd like you to remove regardless: arcadedb.ha.quorumTimeout=1000 is 10× lower than the default of 10s. With majority quorum on 4 nodes and bulk inserts of 50k records, GC pauses or page-flush pauses can easily cross 1s and cause the leader to abort transactions, which then triggers retries and can amplify the symptom. Please drop that override and re-test.

[mdre]

Ok. I will made a clear test from the scratch.
I have an issue with the scheme that I don't understand why it happend. It throw an error on the creation of an index when you run the full script, but if you run that line alone it work well.
Could I send the schema to your mail? I don't wan't to public post it here.

[lvca]

Sure, you can send it to support at arcadedb dot com.

[lvca]

Please retest with the latest main just pushed.

[mdre]

Well. I reinstall a fresh pull in the cluster and rm all databases and remove the line about timeout from the start script.
The issue still there.

1. Start the cluster.
2. Shutdown nodo4.
3. Wait until nodo1 start to claim for nodo4 with a warning every few seconds.
4. Start nodo4.

Nodo1 show
2026-05-06 08:02:42.027 WARNI [GrpcLogAppender] arcadedb-nodo1-test_2434@group-2454BA09CFD7->arcadedb-nodo4-test_2434-AppendLogResponseHandler: received INCONSISTENCY reply with nextIndex 0, errorCount=65, request=AppendEntriesRequest:cid=3637,entriesCount=0

every 5s o 6s.

[lvca]

@mdre — we tracked this down to a real bug in Apache Ratis (the Raft library ArcadeDB's HA layer is built on), not in our code on top of it.

Upstream issue: https://issues.apache.org/jira/browse/RATIS-2523

The short version: on an idle cluster, when a follower restarts with empty Raft storage, the leader's GrpcLogAppender gets stuck in a heartbeat-only INCONSISTENCY reply loop. The follower truthfully tells the leader "my log is empty, send from index 0", and Ratis's LogAppenderBase.getNextIndexForInconsistency ignores the hint in favour of a stale matchIndex+1 — exactly the received INCONSISTENCY reply with nextIndex 0, errorCount=N, ..., entriesCount=0 line you've been seeing every 5-6 s. The bug only fires when the appender has only heartbeats to send (idle cluster); a single user transaction's real AppendEntries breaks the loop, which is why busy clusters never see this. We have a deterministic reproducer (RaftIdleReplicaRestartIT) that exercises your exact 4-step scenario in-process on a single machine.

Workaround landed on main: since 3.2.2 is the latest Apache Ratis release, we shipped a drop-in replacement for GrpcLogAppender that fixes the bug locally. Three small classes under ha-raft/src/main/java/com/arcadedb/server/ha/raft/ratis/ — FixedGrpcLogAppender, FixedGrpcFactory, FixedGrpcRpcType — wired in via RaftPropertiesBuilder. It's a one-line revert away from the stock Ratis once they release the fix.

Could you please pull the latest main, blow away the databases on all 4 VMs (just to be safe), and run your same 4 steps:

1. start the cluster
2. shutdown nodo4
3. wait until nodo1 starts logging the "claiming for nodo4" warnings
4. start nodo4

The leader should converge silently within a handful of seconds and the errorCount warning cadence should stop. If you still see it, capture nodo1's logs from steps 3-4 onwards and share them — I want to make sure no other variant of the same bug remains in your environment.