synchrony doesn't work after upgrading

[Netset]

I have uppgrade from

atlassian-confluence-7.4.0   => atlassian-confluence-7.6.0

I could run confluence but the problem with synchrony:

logs/grep:2020-07-24 14:10:36,431 INFO [http-nio-8090-exec-6] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] retrievePublicKey [Collab editing plugin] Could not retrieve public key for real-time collaboration service at http://localhost:8091/synchrony/jwt-key with exception: Connect to localhost:8091 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

I use apache as reverse proxy.

[Netset]

I can reach localhost:8090 yes.. but even there I can't restart synchrony!

[Shannon S]

Hello, @Netset!

I'm sorry to hear your upgrade hasn't gone so smoothly. Could you tell me if this works outside of your reverse proxy?

The error you're countering is the one from the following article: 

• Confluence could not retrieve public key for real-time collaboration service from Synchrony 

Cause

This is caused by blocking TCP port 8091 in the ELB Security group. This prevents Confluence nodes from reaching Synchrony nodes on 8091. 

You may still be able to reach the Synchrony node directly from the Confluence node on port 8091, but communication occurs through the load balancer (see diagram on this page). 

Can you try the steps in that article, and let me know if you have any trouble?

Take care,

Shannon

[Netset]

Thanks for answering me. But how can I do that? I am using apache https as reverse proxy... am I using ELB??

[Netset]

I can reach localhost:8090 yes.. but even there I can't restart synchrony!

[Shannon S]

Hello,

I recommend that you bypass your proxy, and see if that makes a difference for you. That will verify whether or not the reverse proxy is causing the issue.

In regards to the article I shared with you, you may need to contact your network administrator to make those changes. They need to ensure that port 8091 is not being blocked. They know your network and setup well enough to make the required changes. 

Let me know if you have any questions about that.

Shannon

[Netset]

I have this configuration in server.xml

<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https" proxyName="onlineguide.netset.com" proxyPort="443"/>

<Engine name="Standalone" defaultHost="localhost" debug="0">
<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">
<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
username="name"
password="name"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://127.0.0.1:3305/confluence?useUnicode=true&amp;characterEncoding=utf8"
maxTotal="90"
maxIdle="20"
defaultTransactionIsolation="READ_COMMITTED"
validationQuery="Select 1"/>
<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
<Manager pathname=""/>
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"
reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/>
</Context>
</Host>
</Engine>

I have disabled the apache configuration so I don't have reverse proxy. 

The port 8091 is not blocked in the server..Synchrony still doesn't work

[Netset]

I have changed in /home-directory/bin/setenv.sh to increase heap dump for synchrony:

CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.memory.max=2g ${CATALINA_OPTS}"

but I still see that heap size= 1gb

[Netset]

I have done this: ssh -N -L 9998:localhost:8090 cyg17.netset.com

then navigated to http://localhost:8090 I could reach conflunce server without problem.. I think confluence doesn't start totally, because it doesn't produce 8091 port when I netstat -pnlt | grep 8091 I find nothing

[Shannon S]

In that case, you should shut down Confluence completely, and start it again, without using the proxy, while tailing the Server logs.

Then you'll be able to see if there are any problems that it runs into in order to fix this issue. The start sequence should go from "Starting Confluence..." to "Ready to Serve."

I would also recommend going over these articles:

• Collaborative Editing and Synchrony Troubleshooting 
• Troubleshooting Collaborative Editing 

Once you've had a chance to go through that and get a look at your full stack trace,  please feel free to share with us what you found, and we can try to help you fix the issue.

Take care,

Shannon

[Netset]

Hej Shannon.. I have created a test server

With exactly the same configuration.

The problem is that synchrony doesn't start when ia have httpd enabled and started (I mean the reverse proxy)

[Netset]

what should I do to make synchrony working with reverse proxy.. 

i can send my apache configuration if you want

I ahve repported a bug in confluence 7.6.x

I have reproduced the problem on another server

[Shannon S]

Hello,

Thank you for following-up with the results of your tests.

I'm still confused about one thing, and I just need your clarification on this. I asked you earlier if Synchrony starts up properly if you are not using a reverse proxy at all. 

You mentioned:

I have disabled the apache configuration so I don't have reverse proxy. 

The port 8091 is not blocked in the server..Synchrony still doesn't work

However, your latests updates say that it still doesn't work with reverse proxy. Before you troubleshoot your reverse proxy, you need to make sure that Synchrony starts without any proxy at all. Could you confirm with me whether or not you were able to get Synchrony started WITHOUT your reverse proxy?

Secondly, you mentioned that you reported a bug on Confluence 7.6.x. Do you mean you already reported a bug for this exact issue? I tried looking for it, but I wasn't able to find the bug you raised. Would you please be able to provide that to me?

As far as I can find, there are no specific bugs about Synchrony not starting with reverse proxy, but there is one related to Synchrony not starting when you're using a datasource to connect to Confluence:

• CONFSERVER-60120 Synchrony not starting with datasource after upgrade to Confluence 7.5.2 , 7.6.0 , 7.6.1 & 7.6.2

The workaround is as follows:

Connect Confluence to database using a JDBC connection by specifying DB properties in confluence.cfg.xml file

For Confluence Enterprise Release customers as of 31/07/2020, upgrading to a later unaffected version will also fix the issue. This option is currently not available for customers running 7.5.2+

I found this to be the case for some other customers who thought the issue was related to their reverse proxy.

If this doesn't help, then I will need you to rule out the issues with your reverse proxy configuration first. We want to make sure the issue is actually with Confluence and not your reverse proxy configuration.

Shannon

[Netset]

Could you confirm with me whether or not you were able to get Synchrony started WITHOUT your reverse proxy?

1- Now, I have confluence server on two CentOS servers. I created new server for test purposes. Firstly, I installed confluence 7.4 without reverse proxy (Apache wasn't installed at all on the server). In this case synchrony works fine.

2- Then I upgraded confluence server from 7.4 to 7.6.x where x is - and 1. (Apache still not installed on the server). In this case synchrony doesn't start.

3- Then I downgraded confluence server from 7.6 to 7.4 and installed Apache (reverse proxy). In this case confluence and Synchrony works fine.

4- Then I upgraded confluence server again from 7.4 to 7.6.x. In this case Confluence works but Not synchrony. 

Would you please be able to provide that to me?

https://community.atlassian.com/t5/Confluence-discussions/atlassian-confluence-7-6-X-BUG/m-p/1445359

[Shannon S]

Hello,

Thank you for confirming that. In that case, it does sound like you're affected by the bug I shared with you last week. I'll share that again for reference here:

• CONFSERVER-60120 Synchrony not starting with datasource after upgrade to Confluence 7.5.2 , 7.6.0 , 7.6.1 & 7.6.2
  

You can see here that 7.6.0, 7.6.1, and 7.6.2 are affected, but not 7.6.3 and 7.6.4. In addition, a fix is in progress. There will be more information on this in the next week or so on the bug ticket.

Would you please be able to provide that to me?

Could you let me know what you're looking for here? Is there anything particular I can provide to you?

Take care,

Shannon

[Netset]

Hi,

is there a new LTS release without this bug now? 7.4.3 is in the list with this problem. 

[Shannon S]

Hello,

Confluence 6.13 is a Long Term Support Release. Version 6.13.15 has been released today and resolves this issue. See Issues resolved in 6.13.15 for that information.

Also, quoted from CONFSERVER-60120:

A fix for this issue is available to Server and Data Center customers in Confluence 6.13.15
Upgrade now or check out the Release Notes to see what other issues are resolved.

If you're running the Confluence 6.13 Enterprise release, a fix for this issue is now available in Confluence 6.13.15, which you can find in the Download Archives.

Let me know if you have any questions about this!

Take care,

Shannon

[Netset]

I am using 7.4 Long term support confluence server now. Is that upgrading or downgrading to install 6.13.15?

[Shannon S]

Hi @Netset,

6.13.15 is the recently released LTS version with the fix, but I don't advise that you roll back from 7.4.

The following LTS version we expect to have the fix is 7.4.4. However, this has not yet been released. You can keep an eye out for that release in the near future.

Otherwise, the non-LTS version 7.7.2 has the fix.

If you have any questions or concerns about this, please let me know.

Take care,

Shannon