support for Hazelcast client-server topology user-code-deployment not working

[michaelhaessig]

Hi everyone

im in the process of updating from spring session 1, we have been using the client-server setup for quite some time in production.

Now with the new EntryProcessor implementation of HazelcastSessionRepository i cannot get it to work with the code deployment sample described in the issue #1101 .

i've enabled code deployment both on the server and on the client via the following xml configurations.

Server:

<user-code-deployment enabled="true">
    <class-cache-mode>ETERNAL</class-cache-mode>
    <provider-mode>LOCAL_AND_CACHED_CLASSES</provider-mode>
</user-code-deployment>

Client:

<user-code-deployment enabled="true">
    <classNames>
        <className>org.springframework.session.Session</className>
        <className>org.springframework.session.MapSession</className>
    <className>org.springframework.session.hazelcast.SessionUpdateEntryProcessor</className>
    </classNames>
</user-code-deployment>

I know this config is working because it triggers a new downstream Error when hazelcast is trying to send the SessionUpdateEntryProcessor class to the server.

The Hazelcast server throws the following error:

repository.SessionUpdateEntryProcessor.process(SessionUpdateEntryProcessor.java:27)
hazelcast_1  | 	at com.hazelcast.map.impl.operation.EntryOperator.process(EntryOperator.java:318)
hazelcast_1  | 	at com.hazelcast.map.impl.operation.EntryOperator.operateOnKeyValueInternal(EntryOperator.java:181)
hazelcast_1  | 	at com.hazelcast.map.impl.operation.EntryOperator.operateOnKeyValue(EntryOperator.java:170)
hazelcast_1  | 	at com.hazelcast.map.impl.operation.EntryOperation$EntryOperationOffload$2.run(EntryOperation.java:343)
hazelcast_1  | 	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
hazelcast_1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
hazelcast_1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
hazelcast_1  | 	at java.lang.Thread.run(Thread.java:748)
hazelcast_1  | 	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
hazelcast_1  | 	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
hazelcast_1  | Caused by: java.lang.ClassNotFoundException: Failed to load class org.springframework.security.core.context.SecurityContextImpl from other members.
hazelcast_1  | 	at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:157)
hazelcast_1  | 	at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
hazelcast_1  | 	at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentService.handleClassNotFoundException(UserCodeDeploymentService.java:89)
hazelcast_1  | 	at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentClassLoader.loadClass(UserCodeDeploymentClassLoader.java:57)
hazelcast_1  | 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
hazelcast_1  | 	at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)
hazelcast_1  | 	at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:252)
hazelcast_1  | 	at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:646)
hazelcast_1  | 	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)
hazelcast_1  | 	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)
hazelcast_1  | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)
hazelcast_1  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
hazelcast_1  | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
hazelcast_1  | 	at java.util.HashMap.readObject(HashMap.java:1409)
hazelcast_1  | 	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
hazelcast_1  | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hazelcast_1  | 	at java.lang.reflect.Method.invoke(Method.java:498)
hazelcast_1  | 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
hazelcast_1  | 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)
hazelcast_1  | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
hazelcast_1  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
hazelcast_1  | 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
hazelcast_1  | 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
hazelcast_1  | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
hazelcast_1  | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
hazelcast_1  | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
hazelcast_1  | 	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:82)
hazelcast_1  | 	... 15 more

I've tried to add spring-security jars to the hazelcast server, this fixed this error but triggered more downstream dependency errors.

Does anyone know how to fix this error ?
I've looked at the issues #1312 #1131 maybe they can fix my issue and the need to upload code to the hazelcast server.

To fix my situation i had to completely replace the HazelcastSessionRepository with my own implementation that uses the old way of get/put on the Hazelcast IMap. This is working for now.

Im looking forward to any suggesstions.

[vpavic]

Thanks for the report @michaelhaessig.

As noted in #1101, we've got an integration test for Hazelcast's client-server topology that uses code deployment feature and it works for us without such problems. Can you provide a minimal sample that could be used to reproduce this problem?

[michaelhaessig]

yes i've seen this test. i will try to find time this week to build a sample project.

[michaelhaessig]

hi @vpavic i finally had time to create a sample project for the issue.

here is the link to the repo

the readme should describe the setup. let me know if you have any questions.

[vpavic]

Thanks for providing the sample @michaelhaessig, it's much appreciated!

Unfortunately, it turns out that in client-server topology we still need 3rd party classes that we store in the session (such as Spring Security in your case) present on the server's classpath. Our tests in this scenario only tested the initial save of the sessions, where IMap#set is used, rather then subsequent update of the session, where SessionUpdateEntryProcessor comes into play.

We'll need to further improve the support for client-server topology by implementing the changes proposed in #1131.

With that, this becomes a duplicate of #1131 - please track that one and feel free to vote on it and add further related comments there.

[vpavic]

Duplicate of #1131

[michaelhaessig]

Thanks for the fast feedback. Thats exactly what my investigation brought up too, and with a custom UserDetails object it required even more dependencies to our codebase on the hazelcast server.

I temporarily resolved the situation by using a custom HazelcastSessionRepository implementation which does not use any EntryProcessors by just using IMap#put to update the Map directly.

It would be nice if this could be configured via a flag. The current HazelcastSessionRepository is not flexible to extend. I currently coppied most of its code.

[consult-kk]

@michaelhaessig - I am facing the same exceptions while porting my application from SpringBoot 1.4.5 to 2.1.x .. Can you please share the work around .. do I need to include additional libraries in the classpath on the Hazelcast server and the client app ( in this case its a zuul server with oauth2 )
thanks for your help.
KK

[michaelhaessig]

@consult-kk - sorry for the delayed reply. as mentioned i did replace the full HazelcastSessionRepository to use the old logic from 1.0 with IMap#put instead of using EntryProcessors. This way no libraries are needed on the hazelcast server. If you need the full code let me know.

[flacris]

Hi @michaelhaessig, I am still facing the same problem with a custom UserDetails object. Hazelcast seems to be unable to load (private) nested classes (specifically the AuthorityComparator inside the User class). This is making it impossible to add User Authentication with Spring Security to my application.
I am currently using Hazelcast 5.2.1, Spring Boot 3.0.0, Spring Session 3.0.0 and Spring Security 6.0.0. The cluster is running in a client-server-topology inside a docker container.
Do you still have the code to your custom session repository and would mind sharing it?
Also any help regarding workarounds to the user code deployment issues would be much appreciated!

[michaelhaessig]

Hi @flacris , it has been a while since i mangaed this code :) As mentioned i had to create a full copy of the HazelcastSessionRepository.

i've attached the implementation here: hazelcast-session-repository.zip

Basically what you can see is the property "useEntryProcessor" in "CustomHazelcastSessionRepository" is set to false to not use the hazelcast server side entry processors which avoid having to deploy the full classes to an external hazelcast cluster.

The "CustomSessionUpdateEntryProcessor" is just required to make the implementation complete (but it's not used when useEntryProcessor=false)

Keep in mind this code is using older versions of spring/hazelcast.

[flacris]

@michaelhaessig Thanks a lot for the code!

Unfortunately, in the newest version of spring-session-hazelcast the use of HazelcastIndexedSessionRepository is deeply rooted in the Hazelcast configuration classes. This makes replacing the SessionRepository more complicated than anticipated.

I will keep trying to find a way and hope that Hazelcast resolve their issues with user code deployment and offer better support for the client-server topology!