Community
Products
Confluence
Questions
How to run Confluence Clustered in Kubernetes

How to run Confluence Clustered in Kubernetes

Having issues with running/operating confluence in clustered mode.

Running with Bigbang, istio is the service mesh, and metallb is the load balancer

Helm Chart Version: 1.13.0

Confluence Application Version: 7.19.9

Helm chart values:

values:
  image:
    repository: docker.io/atlassian/confluence
  serviceAccount:
    clusterRole:
      create: true
    clusterRoleBinding:
      create: true
  confluence:
    additionalEnvironmentVariables:
      - name: ATL_FORCE_CFG_UPDATE
        value: "true"
      - name: "http_proxy"
        value: "REDACTED"
      - name: "https_proxy"
        value: "REDACTED"
      - name: "no_proxy"
        value: "REDACTED"
    clustering:
      enabled: true
    forceConfiguration: true
    additionalJvmArgs:
      - -Dcom.redhat.fips=false
      - -Djava.net.preferIPv4Stack=true
      - -Datlassian.dev.mode=false
      - -Datlassian.plugins.enable.wait=300
  podAnnotations:
    traffic.sidecar.istio.io/excludeOutboundPorts: "80,443,5701"
  volumes:
    localHome:
      persistentVolumeClaim:
        create: true
        resources:
          requests:
            storage: 2Gi
      mountPath: "/var/atlassian/application-data/confluence"
    sharedHome:
      nfsPermissionFixer:
        imageRepo: docker.io/library/alpine
        imageTag: 3.18.2
      persistentVolumeClaim:
        storageClassName: ceph-filesystem
        create: true
        resources:
          requests:
            storage: 2Gi
      mountPath: "/var/atlassian/application-data/shared-home"

My initial pod comes up fine. I can step through the setup process, and login as the admin account. And I can see the first node under Clustering.

However when I scale up replicas to 2 nodes. The next pod hangs, and is unable to start the bootstrap with a null pointer exception:

ERROR [Catalina-utility-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
null
java.lang.NullPointerException
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at com.atlassian.confluence.impl.util.db.SingleConnectionDatabaseHelper.getConnection(SingleConnectionDatabaseHelper.java:41)
at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.getBootstrapData(DefaultBootstrapDatabaseAccessor.java:46)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:872)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-10-31 18:36:05,708 WARN [Catalina-utility-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Not starting full plugin system due to upgrade
2023-10-31 18:36:05,716 [Catalina-utility-1]
[Filter: profiling] defaulting to off [autostart=false]
2023-10-31 18:36:06,382 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
31-Oct-2023 18:36:06.396 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
31-Oct-2023 18:36:06.415 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [7960] milliseconds
2023-10-31 18:36:07,859 ERROR [http-nio-8090-exec-1] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter

I have even tried an alternative setup, where I scale the pods before running setup.
If I do this, then all the pods stand up with no errors.
I can step through the setup process, however when I check "Clusters" it shows only 1 node. Additionally, If I scale back down to 0, and then scale back up to 1 or more.
Then all my confluence pods run into the null pointer exception during bootstrapping.

3 answers

0 votes

try to disable cluster authentication in shared cfg.xml

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Further update on the issues I am experiencing.
As long as I have clustering enabled in my helm chart. Any time a reboot confluence, or scale down to 0, then back to 1, confluence is running into the same bootsrapping error everytime.

This is even happening on my primary pod. This only occurs after I have gone through setup steps, trial license, admin user setup, etc...

I am experiencing this issue with helm chart versions as old as 1.13.0, to new as latest, aka 1.16.6

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@Thomas Langhorne it'd be great if you can share the complete atlassian-confluence.log for the 2nd node.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2

INFO:root:Generating /var/atlassian/application-data/confluence/confluence.cfg.xml from template confluence.cfg.xml.j2

INFO:root:User is currently root. Will change directory ownership and downgrade run user to confluence

INFO:root:User is currently root. Will downgrade run user to confluence

INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']

executing as current user

If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

---------------------------------------------------------------------------

Using Java: /opt/java/openjdk/bin/java

2023-10-31 19:35:53,799 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required

---------------------------------------------------------------------------

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-10-31_19-35-53.log instead.

31-Oct-2023 19:35:54.449 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

31-Oct-2023 19:35:54.460 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [proxyPort] to []

31-Oct-2023 19:35:54.468 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]

31-Oct-2023 19:35:54.472 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]

31-Oct-2023 19:35:54.511 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

31-Oct-2023 19:35:54.575 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context/Valve] failed to set property [resolveHosts] to [false]

31-Oct-2023 19:35:54.577 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

31-Oct-2023 19:35:54.854 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

31-Oct-2023 19:35:54.870 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [556] milliseconds

31-Oct-2023 19:35:54.877 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]

31-Oct-2023 19:35:54.877 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.73]

31-Oct-2023 19:35:56.463 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

31-Oct-2023 19:35:56.531 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath

31-Oct-2023 19:35:56.762 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'

2023-10-31 19:35:57,446 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.19.9 [build 8804 based on commit hash d47376fcfbd30b32890091b1ace5511f4b0beb75] - synchrony version 5.0.9

2023-10-31 19:36:00,735 WARN [Catalina-utility-1] [security.java8.serialfilter.DeserializationFilterConfigurator] configure Class sun.misc.ObjectInputFilter not available in classpath. Not setting JDK 8 DeserializationFilter.

java.lang.ClassNotFoundException: sun.misc.ObjectInputFilter

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1412)

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Class.java:315)

at com.atlassian.security.java8.serialfilter.DeserializationFilterConfigurator.configure(DeserializationFilterConfigurator.java:26)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at com.atlassian.confluence.setup.serial.ReflectiveFilterConfigurator.setReflectively(ReflectiveFilterConfigurator.java:11)

at com.atlassian.confluence.setup.serial.JDKFilterConfigurator.setJDKSerialFilter(JDKFilterConfigurator.java:23)

at com.atlassian.confluence.setup.DeserializationFilterConfigurator.configureDeserializationFilter(DeserializationFilterConfigurator.java:33)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:233)

at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.base/java.lang.Thread.run(Thread.java:829)

2023-10-31 19:36:00,930 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$5 Populating setup configuration if running with Cluster mode...

2023-10-31 19:36:00,961 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$5 Finish Populating setup configuration if running with Cluster mode

2023-10-31 19:36:01,038 ERROR [Catalina-utility-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):

null

java.lang.NullPointerException

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Class.java:315)

at com.atlassian.confluence.impl.util.db.SingleConnectionDatabaseHelper.getConnection(SingleConnectionDatabaseHelper.java:41)

at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.getBootstrapData(DefaultBootstrapDatabaseAccessor.java:46)

at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:872)

at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)