Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,293,343
Community Members
 
Community Events
165
Community Groups

how to add a node in cluster (kubernetes env)

Hello Community! Could someone please point me to the right direction on how to add a node to confluence in kubernetes. I've installed it in mono by helm chart with values.yml tuned to my environment. local-home is on pv/pvc (readwriteonce), shared home is on readwritemany. database is postgres in clustering mode, works ok. confluence starts ok without errors then i configure it then start 2nd pod by scaling stateful set. 2nd pod starts ok but how to point it to the first one? tried replacing config the same as 1st one (with and without changing server.id) - won't work. is there a document or git repo with a correct one? please. :) running out of time and ideas. thank you!!

1 answer

1 accepted

0 votes
Answer accepted
Yevhen Atlassian Team Apr 26, 2022

@Alexander Tchaikovski can you please elaborate on "how to point it to the first one"? Confluence uses Hazelcast K8s discovery and when you scale up and the 2nd pod starts without errors and passes a health check, things should happen automatically for you and a new member should join the Hazelcast cluster.

Sharing logs of both pods will help too.

@Yevhen hello. my experience is too short to discuss it deeper. just walking in the darkness. here're 2 logs downloaded from k8s. not found attach file here so will past here: 

pod-0.log:

26-Apr-2022 16:42:45.212 INFO [http-nio-8090-exec-39] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:45.716 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:45.948 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
26-Apr-2022 16:42:45.952 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
26-Apr-2022 16:42:45.952 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
26-Apr-2022 16:42:45.974 INFO [http-nio-8090-exec-29] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:47.564 INFO [http-nio-8090-exec-14] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:47.661 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:47.670 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
26-Apr-2022 16:42:47.670 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
26-Apr-2022 16:42:47.671 INFO [http-nio-8090-exec-14] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
26-Apr-2022 16:42:47.822 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:47.983 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
26-Apr-2022 16:42:47.984 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
26-Apr-2022 16:42:47.984 INFO [http-nio-8090-exec-8] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
26-Apr-2022 16:42:47.990 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.grammars.WadlGrammarsAdaptor
26-Apr-2022 16:42:48.145 INFO [http-nio-8090-exec-6] com.sun.jersey.api.wadl.config.WadlGeneratorLoader.loadWadlGenerator Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport
26-Apr-2022 16:42:48.387 INFO [http-nio-8090-exec-37] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:43:09.172 INFO [http-nio-8090-exec-9] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:43:09.346 WARNING [http-nio-8090-exec-9] com.sun.jersey.spi.inject.Errors.processErrorMessages The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public javax.ws.rs.core.Response com.atlassian.confluence.tinymceplugin.rest.DraftsResource.getDrafts(int,int), should not consume any entity.
26-Apr-2022 16:43:19.576 INFO [http-nio-8090-exec-10] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:43:36.868 INFO [http-nio-8090-exec-36] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:43:49.577 INFO [http-nio-8090-exec-8] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:51:17.085 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:51:22.197 INFO [http-nio-8090-exec-29] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:51:40.631 INFO [http-nio-8090-exec-36] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
26-Apr-2022 16:51:42.497 INFO [http-nio-8090-exec-6] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
27-Apr-2022 03:34:32.372 INFO [http-nio-8090-exec-37] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
27-Apr-2022 03:34:32.391 INFO [http-nio-8090-exec-23] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
27-Apr-2022 03:34:32.453 INFO [http-nio-8090-exec-7] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
27-Apr-2022 03:34:32.701 WARNING [http-nio-8090-exec-7] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://localhost:54350/rest/feature-discovery/1.0/discovered/com.atlassian.confluence.plugins.confluence-dashboard/transition-saved-for-later, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
27-Apr-2022 03:37:13.566 WARNING [http-nio-8090-exec-10] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://wiki.yc.hh/rest/feature-discovery/1.0/discovered/com.atlassian.confluence.plugins.confluence-dashboard/transition-saved-for-later, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expect

pod-1.log:

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/java/openjdk/bin/java
2022-04-26 16:47:07,996 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.000s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2022-04-26_16-47-08.log instead.
26-Apr-2022 16:47:08.333 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]
26-Apr-2022 16:47:08.345 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [proxyPort] to []
26-Apr-2022 16:47:08.355 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]
26-Apr-2022 16:47:08.360 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]
26-Apr-2022 16:47:08.389 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
26-Apr-2022 16:47:08.409 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context/Valve] failed to set property [resolveHosts] to [false]
26-Apr-2022 16:47:08.410 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]
26-Apr-2022 16:47:08.626 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
26-Apr-2022 16:47:08.645 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [424] milliseconds
26-Apr-2022 16:47:08.651 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
26-Apr-2022 16:47:08.651 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.58]
log4j:ERROR Could not find value for key log4j.appender.luceneQuery
log4j:ERROR Could not instantiate appender named "luceneQuery".
26-Apr-2022 16:47:09.565 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
26-Apr-2022 16:47:09.709 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath
2022-04-26 16:47:09,879 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.13.5 [build 8703 based on commit hash da3c91d58b49066fe2f0052e452d9f35d99cdd6f] - synchrony version 4.0.0-master-bfb398fb
26-Apr-2022 16:47:09.888 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'
16:47:09 [Catalina-utility-2] o.a.c.c.C.[.[.[/synchrony-proxy][INFO] log - Initializing Spring DispatcherServlet 'dispatcher'
2022-04-26 16:47:11,783 WARN [Catalina-utility-1] [com.atlassian.config.ApplicationConfig] setApplicationHome Application home does not exist. Creating directory: /var/atlassian/application-data/confluence/shared-home
2022-04-26 16:47:11,787 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Writing setup configuration into shared home...
2022-04-26 16:47:11,794 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$saveSetupConfigIntoSharedHome$9 Finished writing setup configuration into shared home
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/org.apache.felix.framework-5.6.12.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
26-Apr-2022 16:47:17.531 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
26-Apr-2022 16:47:17.540 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [8895] milliseconds
Yevhen Atlassian Team Apr 26, 2022

@Alexander Tchaikovski in your values.yaml do you have `confluence.clustering.enabled: true`? I don't see anything Hazelcast related in the logs.

 

For example, my logs have this:

2022-04-27 06:16:58,725 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] memberAdded [10.6.21.43]:5701 joined the cluster
2022-04-27 06:16:58,726 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] logClusterMembers Cluster now has 2 members: [[10.6.21.103]:5701, [10.6.21.43]:5701]
2022-04-27 06:16:59,739 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Confluence cluster node identifier is [36bcb17f]
2022-04-27 06:16:59,751 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Confluence cluster node name is [confluence-1]

i've managed to start the thing in cluster mode. pod-0 and pod-1 are there in config. one pod is marked in bold, i believe it's correct. there's another issue though. synchrony is not starting. reports:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/atlassian/confluence/confluence/WEB-INF/packages/synchrony-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/atlassian/confluence/confluence/WEB-INF/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Starting synchrony
Exception in thread "main" clojure.lang.ExceptionInfo: error while starting state #'synchrony.config/config {:var #'synchrony.config/config}
any idea what it could be?
Yevhen Atlassian Team Apr 27, 2022

Hmmm... you must have hit a known issue with Synchrony. Can you please try this workaround?

thank you @Yevhen . very much! the issue was wrong synchrony url in the chart. i didn't place https:// in front of it. 

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence

Confluence: Where work and wellness meet

Feeling overwhelmed by the demands of work and life? With a 25% increase in the prevalence of anxiety and depression worldwide during the pandemic, for most of us, it’s a resounding yes . 🙋‍♀️ ...

733 views 5 21
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you