Netty Leak & App Crash with OutOfDirectMemoryError

[pivotal-david-osullivan]

A sample app which creates a DefaultCloudFoundryOperations, Applications and LogRequest object and queries a given app's logs.

When deployed locally and in a CF container and queried via curl, for example, I see the error: LEAK: ByteBuf.release() was not called before it's garbage-collected., and eventually the container crashes with OutOfDirectMemoryError: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 402653191, max: 419430400).

This appears to be the same 2 symptoms discussed in #806 which were addressed in v2.17.0. I've enabled advanced leak detection on local instance, the output is attached and the hints seem to refer to 'reactor.left.httpCodec' among others.

Versions in use: reactor-core: 3.1.8, reactor-netty: 0.7.8, client: 3.11.0, Buildpack: 4.13

Please let me know if I can provide any more info.

Stack.log

[nebhale]

/cc @smaldini @violetagg

[David-VMware]

Hi Team,

Could you share any advice?

Regards,
David

[violetagg]

@David-Pivotal Did you run it with -Dio.netty.leakDetection.level=paranoid also did you enable reactor.ipc.netty=DEBUG?
Also any chance to use Netty 4.1.29.Final?
Thanks,
Violeta

[pivotal-david-osullivan]

@violetagg The stack.log above was with .level=advanced, please see attached log with .level=paranoid and the Netty debug set.

It is based on reactor-netty 0.7.8 which has Netty 4.1.25. Will there be a reactor-netty release soon to include the latest Netty v4.1.29?

cf-client.log

[violetagg]

@pivotal-david-osullivan yes there will be a release this week. Also track this reactor/reactor-netty#422 , most probably we leak the memory when timeout appears.

[violetagg]

@twoseat @nebhale Do you use Flux/Mono#timeout with Reactor Netty. If yes can you apply in CF Java Client a workaround similar to this one spring-cloud/spring-cloud-gateway#540?
Thanks,
Violeta

[twoseat]

@violetagg I just pushed an update including 0.8.0, is there still value in implementing the workaround or will the update suffice?

[violetagg]

@twoseat you should be OK with 0.8.0, the workaround is not needed there.

[pravinkumarb84]

@violetagg I am not able to find the maven repository to include 0.8.0
similar to
compile group: 'io.projectreactor.ipc', name: 'reactor-netty', version: '0.7.9.RELEASE'

I am able to upgrade only until 0.7.9. Please advice on the same to include 0.8.0

[violetagg]

@pravinkumarb84 The coordinates are io.projectreactor.netty:reactor-netty:0.8.0.RELEASE, but CF Java Client needs adaptation in order to use this version. Reactor Netty has changes in the API for creating and configuring the client.