Opentracing-jdbc does not work with quarkus 2.0.0.CR3

[lenalebt]

Describe the bug

When using the dependencies

  implementation("io.quarkus:quarkus-smallrye-opentracing")
  implementation("io.opentracing.contrib:opentracing-jdbc")

with the configuration

quarkus:
  datasource:
    db-kind: "postgresql"
    username: "foobar"
    password: "foobar"
    jdbc:
      url: "jdbc:tracing:postgresql://localhost:5432/foobar"
      driver: "io.opentracing.contrib.jdbc.TracingDriver"

I get the following error logs in the console, and the application does not work:

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
11:46:34.991 WARN  traceId=, parentId=, spanId=, sampled= [io.qu.config] (main) Unrecognized configuration key "quarkus.oidc.credentials.client.secret.value" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
11:46:34.994 INFO  traceId=, parentId=, spanId=, sampled= [or.fl.co.in.li.VersionPrinter] (main) Flyway Community Edition 7.9.1 by Redgate
11:46:35.002 ERROR traceId=, parentId=, spanId=, sampled= [io.qu.ru.Application] (main) Failed to start application (with profile dev): com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface java.sql.Wrapper, interface org.postgresql.PGConnection, interface java.sql.Connection, interface org.postgresql.core.BaseConnection, interface java.lang.AutoCloseable] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
        at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:87)
        at com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass(DynamicProxySupport.java:113)
        at java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:66)
        at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:1006)
        at io.opentracing.contrib.common.WrapperProxy.wrap(WrapperProxy.java:67)
        at io.opentracing.contrib.jdbc.TracingDriver.connect(TracingDriver.java:176)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:204)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:490)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:472)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:834)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

It does work in 1.13.7.Final.

Expected behavior

The application should start up and work as expected, no error logs.

Actual behavior

See above logs :).

To Reproduce

I don't have a reproducer yet, maybe you already know what to do? If not, I can provide one. I guess it's just some class registration missing in graalvm for for the case I do have at hand here, which I think should be done by the dependencies already?

Configuration

quarkus:
  datasource:
    db-kind: "postgresql"
    username: "foobar"
    password: "foobar"
    jdbc:
      url: "jdbc:tracing:postgresql://localhost:5432/foobar"
      driver: "io.opentracing.contrib.jdbc.TracingDriver"

Environment:

Output of uname -a or ver

Linux lena-pc 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment GraalVM CE 21.0.0.2 (build 11.0.10+8-jvmci-21.0-b06)
OpenJDK 64-Bit Server VM GraalVM CE 21.0.0.2 (build 11.0.10+8-jvmci-21.0-b06, mixed mode, sharing)

Quarkus version or git rev

2.0.0-CR3

Build tool (ie. output of mvnw --version or gradlew --version)

---

Gradle 7.0

Build time: 2021-04-09 22:27:31 UTC
Revision: d5661e3f0e07a8caff705f1badf79fb5df8022c4

Kotlin: 1.4.31
Groovy: 3.0.7
Ant: Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM: 11.0.10 (GraalVM Community 11.0.10+8-jvmci-21.0-b06)
OS: Linux 5.11.0-18-generic amd64

Additional context

When I remove the dependencies and do not use the tracing, the application does start up and work as expected.

[quarkus-bot]

/cc @Ladicek, @evanchooly, @kenfinnigan

[kenfinnigan]

Are you running in the JVM, as this looks like an error from running in Native with GraalVM instead?

When it worked in 1.13, was that on JVM or Native?

[gsmet]

@lenalebt I think a reproducer would be welcome. Thanks!

[lenalebt]

I added a reproducer here: https://github.com/lenalebt/quarkus-ticket-18033-reproducer

The initial commit is a working example with 1.13.7.Final, latest commit is the update to quarkus 2.0.0.CR3 from 1.13.7.Final. It works in JVM mode for both 1.13.7.Final and 2.0.0.CR3, but it does not in native mode for 2.0.0.CR3 (I used native mode). JVM works for 2.0.0.CR3.

There is a docker-compose.yml to start the environment (with postgres and jaeger) that you can start with docker-compose up, it's already wired for the application. Jaeger UI is available at localhost:8002.

This is the error message from the reproducer application:

 build/code-with-quarkus-1.0.0-SNAPSHOT-runner 
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-06-22 16:05:24,506 INFO  [org.fly.cor.int.lic.VersionPrinter] (main) Flyway Community Edition 7.9.1 by Redgate
2021-06-22 16:05:24,512 ERROR [io.qua.run.Application] (main) Failed to start application (with profile prod): com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface org.postgresql.PGConnection, interface org.postgresql.core.BaseConnection, interface java.sql.Wrapper, interface java.sql.Connection, interface java.lang.AutoCloseable] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
        at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:87)
        at com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass(DynamicProxySupport.java:113)
        at java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:66)
        at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:1006)
        at io.opentracing.contrib.common.WrapperProxy.wrap(WrapperProxy.java:67)
        at io.opentracing.contrib.jdbc.TracingDriver.connect(TracingDriver.java:176)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:204)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:490)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:472)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:68)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:834)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)

If there is anything more I can provide, I'll do that :).

[kenfinnigan]

It definitely looks like some classes were removed from the Native Image that shouldn't have.

However, I'm not sure why it's happening.

[gsmet]

@kenfinnigan it's not complaining about a missing class AFAICS but about a missing GraalVM proxy definition (i.e. with NativeImageProxyDefinitionBuildItem).

I think we will need to add some additional magic to the Agroal extension.

I'll have a look but it's too late for 2.0.0.Final.

[kenfinnigan]

Ah yes, you're right.

[luneo7]

This mainly happened because there was a version bump in the opentracing-jdbc lib in 2.0.0, with the bump from 0.0.12 to 0.2.12, a new class WrapperProxy was added that is doing dynamic proxy...

[loicmathieu]

@luneo7 this change appears to have been added to 0.2.8, can you try with 0.2.7 to see if it works ?
@kenfinnigan maybe we can downgrade to 0.2.7 while waiting to fix the proxy issue ?

We don't have a native test with the opentracing contrib JDBC tracer so we must be careful when upgrading it (and test it manually)!

[luneo7]

I've done some tests... to make it work in native mode I had to do all possible permutations of the interfaces for the PostgreSQL classes (PgConnection, PgStatemet, PgPreparedStatement, PgCallableStatement) to build a proxy config for GraalVM.... WrapperProxy is using Classes.getAllInterfaces and it does not guarantee the order of the returning classes array... so it ended up being a huge proxy-config.json with all possible permutations (https://gist.github.com/luneo7/453ea0bd0d51a48c6c4377237e8ad831), and the build time was hugely increased...

Using the 0.2.7 version doesn't help, just because it has a DynamicProxy class that does the same thing as the WrapperProxy... so the same problem happens... The only "newer" version that is not using dynamic proxy is the 0.2.4 version...

If you change the gradle dependency config to:

[...]
    implementation("io.opentracing.contrib:opentracing-jdbc:0.2.4!!") {
        isForce = true
    }
[...]

It will definitely work... so this is a workaround for Quarkus 2.0.0 for now....

[kenfinnigan]

Thanks for the investigations @luneo7, appreciated!