parameter sweep crashes, GPU memory fills up

[pawlowska]

Hi, I tried running eSRRF on Quadro K2200 GPU. It works in principle (YAY!) but crashes sometimes. For example yesterday it crashed at step 22/50 of the parameters sweep, meaning it did 21 steps just fine. I am including the error below. I also noticed that the GPU memory was showing as nearly full, 3.9GB out of 4GB - can it be that the memory somehow does not get released.....?

(Fiji Is Just) ImageJ 2.3.0/1.53q; Java 1.8.0_172 [64-bit]; Windows 10 10.0; 1763MB of 73552MB (2%) com.jogamp.opencl.CLException$CLMemObjectAllocationFailureException: can not enqueue 1DRange CLKernel [id: 202273553280 name: kernelResetFramePosition] with gwo: null gws: {1} lws: null cond.: null events: null [error: CL_MEM_OBJECT_ALLOCATION_FAILURE] at com.jogamp.opencl.CLException.newException(CLException.java:84) at com.jogamp.opencl.CLCommandQueue.putNDRangeKernel(CLCommandQueue.java:1638) at com.jogamp.opencl.CLCommandQueue.put1DRangeKernel(CLCommandQueue.java:1496) at com.jogamp.opencl.CLCommandQueue.put1DRangeKernel(CLCommandQueue.java:1466) at nanoj.liveSRRF.LiveSRRF_CL.resetFramePosition(LiveSRRF_CL.java:958) at nanoj.liveSRRF.gui.ParametersSweep_.run(ParametersSweep_.java:430) at ij.IJ.runUserPlugIn(IJ.java:243) at ij.IJ.runPlugIn(IJ.java:204) at ij.Executer.runCommand(Executer.java:152) at ij.Executer.run(Executer.java:70) at java.lang.Thread.run(Thread.java:748)

[micahschott]

I also ran into problems today. I'm not really sure how to troubleshoot this.

[HannahSHeil]

Thanks for finding this problem! I'll look into it!

[ammendes]

I also ran into problems today. I'm not really sure how to troubleshoot this.

I think your problem is different from the OP's. Seems like you have a problem with loading OpenCL, either due to a bad installation or missing Java bindings (i.e., JOCL).

[HannahSHeil]

Hi @pawlowska, sorry for taking so long to reply to your request. The memory overflow on NVidia graphics cards has been described before here: https://clij.github.io/clij2-docs/faq. ->> Big Images on NVidia graphics cards

[bheit]

Just to add some more information to this issue. We've run parameter sweep on the exact same acquisition image across 5 computers with odd results.

• 2 computers with NVIDIA cards (P4000 and a GTI 30XX) crashed with the above issues about half way through the run
• 2 computers with AMD cards completed the process (radions, don't know the model), but produced a blank FRC map
• 1 laptop running Intel on-motherboard graphics (Iris Xe) completed the process successfully (if somewhat slowly)

I'm not sure if that will help track down the issue or not, but this is an extensive a test as we can perform.

[HannahSHeil]

Thank you for sharing this info, @bheit!

[HannahSHeil]

Hi
Could you please try the following fix?

Go to the folder /.../Fiji.app/jars
Remove these:
jogl-all-2.4.0-rc-20211011.jar
jocl-all-2.4.0-rc-20211011.jar
joal-all-2.4.0-rc-20211011.jar
gluegen-rt-2.4.0-rc-20210111.jar

Go to: https://jogamp.org/deployment/archive/rc/v2.5.0-rc-20230523/jar/
Download the following files: jogl-all.jar, jocl.jar, joal.jar, gluegen.jar
Place these four files in the folder /.../Fiji.app/jars

Go to the folder /.../Fiji.app/jars/win64
Remove these:
gluegen-rt-2.4.0-rc-20210111-natives-windows-amd64.jar
joal-2.4.0-rc-20210111-natives-windows-amd64.jar
jocl-2.4.0-rc-20210111-natives-windows-amd64.jar

Go to: https://jogamp.org/deployment/archive/rc/v2.5.0-rc-20230523/jar/
Download the following files: gluegen-rt-natives-windows-amd64.jar, jocl-natives-windows-amd64.jar, joal-natives-windows-amd64.jar
Place these three files in the folder /.../Fiji.app/jars/win64

Restart Fiji

[HannahSHeil]

If you are working on Windows 10/11 and you would like to use other NanoJ Plugins aswell you might also need to install the OpenCL™ and OpenGL® Compatibility Pack (https://apps.microsoft.com/store/detail/opencl%E2%84%A2-and-opengl%C2%AE-compatibility-pack/9NQPSL29BFFF?hl=en-us&gl=us&activetab=pivot%3Aoverviewtab)

In that case please make sure to select the following settings: Advanced settings > Processing device > Default device

[bheit]

I have done everything recommended above (installed the windows OpenGL/CL pack, replaced the jar files, and edited the registry) and I am still having the parameter sweep fail. It also fails at the exact same place (frame 58/60), regardless of the size of the base file use.

The error I get is:

(Fiji Is Just) ImageJ 2.14.0/1.54f; Java 1.8.0_322 [64-bit]; Windows 10 10.0; 1122MB of 24347MB (4%)

com.jogamp.opencl.CLException$CLDeviceNotAvailableException: can not create CL context [error: CL_DEVICE_NOT_AVAILABLE]
at com.jogamp.opencl.CLException.checkForError(CLException.java:67)
at com.jogamp.opencl.CLContext.createContextFromType(CLContext.java:204)
at com.jogamp.opencl.CLContext.create(CLContext.java:172)
at com.jogamp.opencl.CLContext.create(CLContext.java:155)
at nanoj.liveSRRF.LiveSRRF_CL.initialise(LiveSRRF_CL.java:179)
at nanoj.liveSRRF.gui.ParametersSweep_.run(ParametersSweep_.java:452)
at ij.IJ.runUserPlugIn(IJ.java:244)
at ij.IJ.runPlugIn(IJ.java:210)
at ij.Executer.runCommand(Executer.java:152)
at ij.Executer.run(Executer.java:70)
at java.lang.Thread.run(Thread.java:750)

[bheit]

Fixed it - for some reason parameter sweep was using the windows renderer instead of the NVIDIA 3060 for processing. I switched the device from 'default' to 'NVIDIA' under the "Advanced" options and it ran correctly (and 5x as fast!)