Potential deadlock situation when Cantaloupe is restarted under heavy load

[taylor-steve]

I can't say yet if this is a Cantaloupe specific issue or a local configuration issue, but @jcoyne encouraged me to share what we've found here.

Description

Our production Cantaloupe servers were sometimes entering a state in which they would respond to health checks and template routes (e.g., /iiif/2) but would hang indefinitely when IIIF requests were made.

Discovery

We finally caught this happening live and the (apparent) timeline looks like:

• Kakadu crashes, taking Cantaloupe with it
• Seeing the failure, systemd restarts Cantaloupe
• Our load balancer hasn't had time to recognize the failure so our reverse proxy is feeding Cantaloupe a flood of requests the moment it comes back up
• Within seconds, Cantaloupe has entered the degraded state. Looking at the output of jstack we see all the qtp threads are in this state or similar:

qtp1177067563-103" #103 [1610475] prio=5 os_prio=0 cpu=246.17ms elapsed=3329.79s tid=0x00007c44e0009970 nid=1610475 in Object.wait()  [0x00007c462d0fc000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.jena.rdf.model.ModelFactory.createDefaultModel(ModelFactory.java:91)
	- waiting on the Class initialization monitor for org.apache.jena.rdf.model.impl.ModelCom
	at edu.illinois.library.cantaloupe.image.Metadata.loadXMP(Metadata.java:232)
	at edu.illinois.library.cantaloupe.image.Metadata.getXMPModel(Metadata.java:202)
	at edu.illinois.library.cantaloupe.image.Metadata.readOrientationFromXMP(Metadata.java:160)

Full log: deadlock-jstack.log

Seeing all of the waiting on the Class initialization monitor for org.apache.jena.rdf.model.impl.ModelCom entries looks like a deadlock to me. We recreated this situation in our stage environment by:

• Using ab (Apache Benchmark) to send a constant stream of info.json requests at a rate consistent with a heavy burst of traffic
• Restarting Cantaloupe

Doing this, Cantaloupe would enter the degraded state reliably, with similar jstack output.

Remediation

Increasing http.min_threads from 8 to 16 has been sufficient so far at making Cantaloupe resilient to this, in our particular situation.

[DiegoPino]

Hi @taylor-steve what version of Java? 20? Maybe similar to this? https://bugs.openjdk.org/browse/JDK-8288064
systemd restarts Cantaloupe or tries to stop it, waits and starts it again?

[taylor-steve]

Hi @DiegoPino. We're on 21 (openjdk 21.0.8 2025-07-15). I definitely haven't ruled out Java specific issues, that's a good thought.

We're only seeing Cantaloupe start once in the logs after the crash. Because we see the same behavior in our stage environment when we restart the service manually I haven't been looking hard at systemd, but I suppose there could be something there.

[DiegoPino]

@taylor-steve Makes sense. We have seen Cantaloupe run out of threads under heavy load too, without restarting and in the middle of operations. It is still there, just not responding to requests and we are running openjdk 21 too (and 23 in some instances too). We run dockerized so we have now a health monitor in our docker compose that hits uncached IIIF/2 URLS and stops (down) and (up) again when that happens

[glenrobson]

Note this was raised here: #228 (comment)

Test when we do the spring update.