GraalVM: Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

[patpatpat123]

Hello team,

This is my first issue in this repo.
Is not anything, big thanks for this very cool project.

I would like to reach out in order to report an issue:
My project is a GraalVM native image project, a simple Kafka consumer.

In my dependencies, I do have the most recent Snappy:

        <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.1.9.1</version>
        </dependency>

On my machine, Ubuntu, I do have Snappy installed.

root@dlp:~# apt -y install snapd
root@dlp:~# systemctl  enable snapd.service snapd.socket
root@dlp:~# source /etc/profile.d/apps-bin-path.sh

Yet, when running the application, I am facing this 100% reproducible issue:

Caused by: org.apache.kafka.common.KafkaException: Received exception when fetching the next record from my-topic-1. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1676)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1497)
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:717)
	at org.apache.kafka.clients.consumer.internals.Fetcher.collectFetch(Fetcher.java:683)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1314)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1243)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
	at reactor.kafka.receiver.internals.ConsumerEventLoop$PollEvent.run(ConsumerEventLoop.java:357)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base@17.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@17.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:46)
	at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:94)
	at org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:276)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:280)
	at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:364)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1619)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1656)
	... 16 common frames omitted
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:343)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:177)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:155)
	at org.xerial.snappy.Snappy.init(Snappy.java:70)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:44)
	... 22 common frames omitted

Could you please help on this issue?

[sgammon]

i am encountering this too -- the native library loads fine on jvm, just not native image

[xerial]

SnappyLoader tries to copy a native lib inside https://github.com/xerial/snappy-java/tree/master/src/main/resources/org/xerial/snappy/native using the code at
https://github.com/xerial/snappy-java/blob/v1.1.9.1/src/main/java/org/xerial/snappy/SnappyLoader.java#L361

It seems the behavior of finding resources from the classpath is different between GraalVM and JVM.

[patpatpat123]

Just checking a month after, anyone can help one this issue please?

[shautvast]

Think this may be of help @patpatpat123
finding resources from the classpath IS different between GraalVM and JVM
see https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/Resources/
you'd have to to supply the native lib to native-image option -H:IncludeResources

but I also see that instead of throwing Exception, snappy could fallback to -Djava.library.path which I don't think it does now

[xerial]

FYI: Currently, I have no idea how I can fix this issue mainly because I'm not a user of GraalVM. So I will not work on this unless there will be a PR. The problem looks like just finding resource files (.so files) from the classpath (or inside the jar file) in GraalVM.

[patpatpat123]

I tried the resource, adding it when running the graalVM native image, and unfortunately, the issue still persists.

I guess I can just cross my fingers for when the fix will be available..

[patpatpat123]

@shautvast

Thank you for your input.

Just to avoid misunderstanding, I should include the -H: IncludeResources=abc at build time or at run time?

Also, what should be the value- H: IncludeResources**=abc** ?

Thank you

[shautvast]

@patpatpat123 I just verified that this is actually quite easy to do at build-time:
just include -H:IncludeResources=".*/libsnappyjava.dylib" in the native-image call
and make sure that the native lib is the correct one for your platform (and of course provide the snappy jar in the classpath for native-image)

so the complete call would look like this
native-image -cp [your_classes]:[PATH]/snappy-java-1.1.9.1.jar -H:IncludeResources=".*/native-lib" MainClass

inspect the snappy-jar to find the native libs

@xerial this is not something that can be fixed/changed from the library side

[xerial]

It seems sqlite-jdbc, which is using the same mechanism for loading native library at runtime, can support GraalVM https://github.com/xerial/sqlite-jdbc#graalvm-native-image-support

Probably we can borrow the same technique

[patpatpat123]

Hey team, I tried this:

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <configuration>
        <buildArgs>
            <buildArg>-H:IncludeResources=".*/native-lib"=${project.build.directory}</buildArg>
        </buildArgs>
    </configuration>
</plugin>

Based on the same example from sqlite. However, it is still not working, still same issue.

May I ask if I did something wrong in my command?
I would like to leverage org.graalvm.buildtools native-maven-plugin if possible

[liuxgo]

@patpatpat123
Write resource-config.json like this, I only need binary file for Windows.

{
  "bundles": [],
  "resources": {
    "includes": [
      {
        "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"
      }
    ]
  }
}

Add build args in pom.xml like this: <buildArg>-H:ResourceConfigurationFiles=native-image/resource-config.json</buildArg>

[pavelorehov]

I have same issue on Linux and could NOT resolve it by adding resource to native image like below.
Also not sure how to verify that indeed libsnappyjava.so is part of native image, may be I'm doing something wrong.

Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

<plugin>
	<groupId>org.graalvm.buildtools</groupId>
	<artifactId>native-maven-plugin</artifactId>
	<configuration>
		<buildArgs>
			<arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
		</buildArgs>							
	<requiredVersion>22.3</requiredVersion>
	</configuration>
	...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {    
        "pattern": "/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

The only workaround I found is to copy the libsnappyjava.so to same folder with executable and pass parameters, but that is pretty dirty fix I guess.
-Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Dorg.xerial.snappy.tempdir=/tmp