An error occurred when running Synchrony in Confluence(docker)

[박경민]

I entered and ran the following command to deploy the confluence cluster on docker.

docker run -d --name="confluence" \
-p 8090:8090 -p 8091:8091 --network=host \
-v /atlassian/confluence:/var/atlassian/application-data/confluence \
-v /atlassian/shared-home/confluence:/atlassian/shared-home/confluence \
-v /atlassian/confluence-custom/etc/seraph-config.xml.j2:/opt/atlassian/etc/seraph-config.xml.j2 \
-e CONFLUENCE_SHARED_HOME=/atlassian/shared-home/confluence \
-e JVM_MINIMUM_MEMORY=1024m \
-e JVM_MAXIMUM_MEMORY=2048m \
-e ATL_PROXY_NAME=eocis.com \
-e ATL_PROXY_PORT=443 \
-e ATL_TOMCAT_SCHEME=https \
-e ATL_TOMCAT_CONTEXTPATH="/wiki" \
-e ATL_CLUSTER_TYPE=tcp_ip \
-e ATL_CLUSTER_NAME=confluence \
-e ATL_PRODUCT_HOME_SHARED=/atlassian/shared-home/confluence \
-e ATL_CLUSTER_TTL=1 \
-e ATL_CLUSTER_PEERS=10.1.128.10,10.1.129.10 \
-e TZ=Asia/Seoul \
-e JVM_SUPPORT_RECOMMENDED_ARGS=-Dconfluence.cluster.node.name="worker-1" \
atlassian/confluence:7.19.5-jdk11

And then I encountered a warn, the contents of which are as follows.
Functions related to synchrony are not normal.

2023-09-12 12:40:10,687 WARN [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] lambda$pollHeartbeat$1 Rescheduled heartbeat check failed.
io.atlassian.util.concurrent.TimedOutException: Timed out after: 30009336411 NANOSECONDS
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyMonitor.lambda$pollHeartbeat$0(DefaultSynchronyMonitor.java:140)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-09-12 12:40:10,688 WARN [synchrony-interop-executor:thread-1] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
2023-09-12 12:40:10,691 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startupSynchrony Starting Synchrony and enabling Collaborative Editing
2023-09-12 12:40:10,954 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.SynchronyEnvironmentBuilder] isSynchronyProxyEnabled proxy port present: true
2023-09-12 12:40:11,126 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.SynchronyProcessBuilder] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2023-09-12 12:40:11,127 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.SynchronyProcessBuilder] debugPrintEnvironment Synchrony command line: /opt/java/openjdk/bin/java -classpath /opt /atlassian/confluence/temp/5.0.3.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/postgresql-42.5.0.jar -Xss2048k -Xmx2g synchrony.core sql
2023-09-12 12:40:11,135 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://eocis.com/synchrony,https:/ /eocis.com/wiki/synchrony-proxy
2023-09-12 12:40:11,136 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://eocis.com/synchrony/v1
2023-09-12 12:40:11,136 INFO [synchrony-interop-executor:thread-7] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1

The apache config is as follows.

<VirtualHost *:80>
DocumentRoot "/var/www/html"
ServerName eocis.com
RewriteEngine On
Redirect "/" "https://eocis.com/"
RemoteIPHeaderX-Forwarded-For
</VirtualHost>

<VirtualHost *:443>
ServerName eocis.com
ProxyRequests Off
RewriteEngine On
ProxyVia Off
RemoteIPHeaderX-Forwarded-For

<Proxy *>
Require all granted
</Proxy>

#LoadbalancerMonitoringPage
<Location/balancer-manager>
SetHandler balancer-manager
Require all granted
</Location>
#LoadbalancerSetup
#Crowd
<Proxy balancer://crowdcluster>
#CrowdNODE 1
BalancerMember http://localhost:8095/crowd route=node1

#LoadbalancerSettings
ProxySet lbmethod=byrequests
ProxySet stickysession=JSESSIONID
</Proxy>
ProxyPass /crowdbalancer://crowdcluster
ProxyPassReverse /crowdbalancer://crowdcluster

#Confluence
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/wiki" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://confluencecluster>
#Confluence NODE 1
BalancerMember http://10.1.128.10:8090/wiki route=node1

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/wiki" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://confluencecluster>
#Confluence NODE 1
BalancerMember http://10.1.128.10:8090/wiki route=node1
# Confluence NODE 2
BalancerMember http://10.1.129.10:8090/wiki route=node2

# Loadbalancer Settings
ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
</Proxy>
ProxyPass /wiki balancer://confluencecluster
ProxyPassReverse /wiki balancer://confluencecluster

Can I get about the problems and improvements?

[Mahesh Shinde]

Hi @박경민 :

I gone through your apache configuration but missing synchrony configuration.

Please go through the article and update you apache config: https://confluence.atlassian.com/confkb/how-to-use-apache-as-a-load-balancer-for-confluence-data-center-970602623.html

Let me know if you still facing this issue.

Regards,

Mahesh Shinde 

[박경민]

I changed it as below, but it remains the same.

<VirtualHost *:80>
DocumentRoot "/var/www/html"
ServerName eocis.com
RewriteEngine On
Redirect "/" "https://eocis.com/"
RemoteIPHeaderX-Forwarded-For
</VirtualHost>

<VirtualHost *:443>
ServerName eocis.com
ProxyRequests Off
RewriteEngine On
ProxyVia Off
RemoteIPHeaderX-Forwarded-For

<Proxy *>
Require all granted
</Proxy>

#LoadbalancerMonitoringPage
<Location/balancer-manager>
SetHandler balancer-manager
Require all granted
</Location>
#LoadbalancerSetup
#Crowd
<Proxy balancer://crowdcluster>
#CrowdNODE 1
BalancerMember http://localhost:8095/crowd route=node1

#LoadbalancerSettings
ProxySet lbmethod=byrequests
ProxySet stickysession=JSESSIONID
</Proxy>
ProxyPass /crowdbalancer://crowdcluster
ProxyPassReverse /crowdbalancer://crowdcluster

# Main Synchrony proxy rule
ProxyPass /synchrony balancer://synchrony-http/synchrony

# Synchrony HTTP balancer declaration
<Proxy balancer://synchrony-http>
BalancerMember http://10.1.128.10:8091 route=synnode1
BalancerMember http://10.1.129.10:8091 route=synnode2

Require all granted
</Proxy>

# Synchrony WebSocket balancer declaration
<Proxy balancer://synchrony-ws>
BalancerMember ws://10.1.128.10:8091 route=synnode1
BalancerMember ws://10.1.129.10:8091 route=synnode2

Require all granted
</Proxy>

# WebSocket rewrite conditions and rules
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/synchrony/(.*)/bayeux-sync1 [NC]
RewriteRule /(.*) balancer://synchrony-ws/$1 [P,L]

#Confluence
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/wiki" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://confluencecluster>
#Confluence NODE 1
BalancerMember http://10.1.128.10:8090/wiki route=node1
# Confluence NODE 2
BalancerMember http://10.1.129.10:8090/wiki route=node2

#LoadbalancerSettings
ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
</Proxy>
ProxyPass /wiki balancer://confluencecluster
ProxyPassReverse /wiki balancer://confluencecluster

#Jira
<Proxy balancer://jiracluster>
# Jira NODE 1
BalancerMember http://10.1.128.10:8080/jira route=node1
#JiraNODE2
BalancerMember http://10.1.129.10:8080/jira route=node2

#LoadbalancerSettings
ProxySet lbmethod=byrequests
ProxySet stickysession=JIRASESSIONID
</Proxy>
ProxyPass /jira balancer://jiracluster
ProxyPassReverse /jira balancer://jiracluster

SSLEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLHonorCipherOrder on

SSLCertificateFile /root/certification/cert.pem
SSLCertificateKeyFile /root/certification/key.pem
SSLCertificateChainFile /root/certification/fullchain.pem

</VirtualHost>

---

And when you press the page edit button, the following error appears and the page cannot be edited.

2023-09-12 15:22:06,781 INFO [http-nio-8090-exec-7] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] retrievePublicKey [Collab editing plugin] Could not retrieve public key for real-time collaboration service at http ://127.0.0.1:8091/synchrony/jwt-key with exception: Connect to 127.0.0.1:8091 [/127.0.0.1] failed: Connection refused (Connection refused)
2023-09-12 15:22:06,784 ERROR [http-nio-8090-exec-7] [confluence.plugins.synchrony.SynchronyContextProvider] getContextMap Could not retrieve Synchrony public key.
-- url: /wiki/pages/editpage.action | page: 1867777 | traceId: 60a65142eb9fff2d | userName: atlassian-admin | referer: https://eocis.com/wiki/ | action: editpage

And the server's port 8091 does not come up.

What on earth is wrong?

[Mahesh Shinde]

Hi @박경민

Apache load balancer and Confluence node is on the same server or on different servers?

TCP port 8091 should be reachable from LB to confluence nodes, Please check this port is enable or not.

Also please make sure below config you have validate on LB.

Prerequisites

1. Apache version 2.4+ with support for support WebSockets
2. The following modules must be enabled in your main httpd.conf file:

   1. proxy_module

   2. proxy_http_module

   3. proxy_wstunnel_module
      
   4. proxy_balancer_module
   5. lbmethod_byrequests_module
      
   6. headers_module
   7. rewrite_module

Regards,

Mahesh Shinde

[박경민]

The apache and confluence docker servers are separate.
The targets 10.1.128.10 and 10.1.129.10 visible in apache config are all docker servers running confluence.

Additionally, the module has been activated using the command below.
a2enmod proxy \
proxy_http\
proxy_wstunnel \
proxy_balancer \
lbmethod_byrequests\
headers\
rewrite \
proxy_hcheck

[박경민]

The server is on AWS, and all ports for confluence, including port 8091, are open in the security group.

However, after executing the previously mentioned docker run command and starting confluence, port 8091 was not open.