SpringMVC dispatcher [springMvc] could not be started

[Holly Marsh]

Upgrading from 7.21.4 to 8.5.4. Using git 2.35.3, and java 17.0.3. Starting up the first node in my Data Center cluster, I'm seeing this in the logs:

launcher.log

[main] DEBUG com.atlassian.bitbucket.internal.boot.security.JmxDeserializationFilterReflectionRegister - Running on Java 9+ using java.io.ObjectInputFilter

atlassian-bitbucket.log

2023-09-23 09:21:16,667 ERROR [hz.hazelcast.event-1] c.h.s.i.e.impl.EventServiceImpl [10.2.39.133]:5701 [bitbucketProd] [3.12.13] hz.hazelcast.event-1 caught an exception while processing:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@1cb6e277
com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.InvalidClassException: filter status: REJECTED
at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:70)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:193)
at com.hazelcast.map.impl.DataAwareEntryEvent.getValue(DataAwareEntryEvent.java:90)
at com.atlassian.cache.hazelcast.HazelcastCacheManager.lambda$new$1(HazelcastCacheManager.java:106)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:67)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:64)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78)
at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227)
Caused by: java.io.InvalidClassException: filter status: REJECTED
at java.base/java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1414)
at java.base/java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1868)
at java.base/java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1821)
at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:839)
at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:988)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2034)
at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1909)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2235)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:84)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
... 15 common frames omitted
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make public java.lang.Class java.io.ObjectInputStream$FilterValues.serialClass() accessible: module java.base does not "opens java.io" to unnamed module @2f112965
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at com.atlassian.bitbucket.internal.boot.security.JmxDeserializationFilterReflectionRegister$DeserializationFilterInvocationHandler.invoke(JmxDeserializationFilterReflectionRegister.java:87)
at jdk.proxy1/jdk.proxy1.$Proxy0.checkInput(Unknown Source)
at java.base/java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1383)
at java.base/java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1868)
at java.base/java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1821)
at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:839)
at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:988)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2034)
at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1909)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2235)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:84)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
at com.hazelcast.map.impl.DataAwareEntryEvent.getValue(DataAwareEntryEvent.java:90)
at com.atlassian.cache.hazelcast.HazelcastCacheManager.lambda$new$1(HazelcastCacheManager.java:106)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:67)
at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:64)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105)
at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78)
at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227)
... 3 frames trimmed

[Kai Günster]

We had exactly the same problem with a fresh install of Bitbucket 8.9.x using Java 17, running on Debian Bookworm. We found a solution, but I must emphasize that it's not a great solution. 

To make it work, extend your JAVA_OPTS environment as follows:

HAZELCAST_OPTS=" --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL- UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMEDi --add-opens java.base/java.io=ALL-UNNAMED"

JAVA_OPTS=" $JAVA_OPTS $HAZELCAST_OPTS"

Your Bitbucket instance should then come up and work.

Now, why is that not a great solution? What these settings essentially do is disable some Java security features that were added in Java 9. Without going into too much detail, before Java 9 and library and plugin could access all the JDK core classes, with Java 9 this was restricted through the use of a new module system that can deny or allow this access. What these settings do is open a lot of core modules again to be used by anyone, probably including Bitbucket plugins. 

This exposes you to the same kind of risks that you used to have with Java 8 and earlier anyway, but they are risks you could avoid now. 

I would consider this problem a Bitbucket bug, IMO the start scripts should set the necessary module permissions, and only those, not the ALLOW ALL TO ALL approach I'm taking here. 

[Kai Günster]

Addendum: with the above solution, our BitBucket instance came up, but then we got problems with LDAP authentication also due to the Java module system. to fix those, we had to extend HAZELCAST_OPTS in the code above to include 

 --add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED

I guess at that point it's not technically accurate anymore to call it HAZELCAST_OPTS, but meh.