unable to create page after upgrade to 6.0.1 with apache 2.2 as proxy

Hi,

after I upgraded to latest 6.0.1 I'm not able to create pages anymore. The loading symbol does not disapear and after some time I see the message:

Loading the editor's taking longer than usual. Give it a few moments, then refresh your page if it still doesn't load. Speak to your Confluence admin if that doesn't fix it.

Refreshing the page was not helpful and I had a look at the catalina log and the atlassian-confluence.log, both do not show an error while loading a "add page".

Do you have any ideas what to try to solve this problem?

Cheers
Andi

8 answers

3 votes

Hi everyone, 

Sorry to hear you've had problems getting collaborative editing up and running with your reverse proxy.  It's important to know your reverse proxy must support websockets. This means Apache 2.2 won't work, you need Apache 2.4.    

The following docs have some example configs that you can use as a basis for your own config.  Note that the order of directives and blocks is important in the examples. 

 

I don't agree with you. Is possible use proxy_wstunnel_module module with 2.2. @Andreas Nitsche wrote down how to get this module, see below

I am running Apache 2.4, but all I needed to make it work was set synchrony.proxy.enabled property to false

set synchrony.proxy.enabled property to false  work for me.

my confluence version is 6.6.1

Hi I just changed one system property and it worked for me.

In file <Confluence home>\confluence.cfg.xml I changed synchrony.proxy.enabled property to false.

Documentation

The problem comes the new collaborative script which surprise surprise is running at your port 8091 by default. So, open the port in the firewall and go to Admin -> Left panel down is the Collaborative Editing -> restart the synchrony.

Oh! And there is a new file called atlassian-synchrony in the log folder of your app data to help you debug.

thanks Panos, this was the solution to my problem. but in my case i've temporary turned off collaborative editing feature and i will check firewall / router problem later wink with our network admin. 

 

Thanks! Worked for me. I just turned off Collaborative Editing (we dont use it)

I am using Nginx to listen 80 and forward requests to localhost 8090,only 80/tcp port opened on firewall, and meet the problem that cannot edit any page. This solution works like charming. Thanks Panos!

I do have an apache in front of my confluence to do the SSL handshake for me. Looks like there is the problem because the OPTION request to port 8091 is also called via https. Is there already a solution documented?

I would start to proxy this service through apache as well. Shrink the listen for port 8091 to 127.0.0.1, setup the proxy on port 8091 for the external ip including ssl and test again. I will try that later or tomorrow.

Cheers
Andi

ah so i am not the only one with problem when ssl

Just an update on how far I am at the moment.

The idea to configure synchrony to localhost was more difficult than I thought at the beginning because the documentation for synchrony.bind and synchrony.port are well hidden. I checked this site to see how I can configure synchrony for data center even if I do not have one: https://confluence.atlassian.com/doc/configuring-synchrony-for-data-center-858772125.html

Once the two options

JAVA_OPTS="$JAVA_OPTS -Dsynchrony.bind=127.0.0.1"
JAVA_OPTS="$JAVA_OPTS -Dsynchrony.port=8091"

are set, synchrony starts at localhost:8091 as expected. After that I configured apache to listen on the external ip on port 8091. Make to to not only use "Listen 8091" but use "Listen x.y.z.a:8091" and if you've enabled IPv6 according "Listen a:b:c::x:8091"

At the moment my collaboration service is NOT working yet, even if the service can be accessed through Apache with SSL. This is because a websocket is used for synchrony and my apache 2.2.15 of the CentOS 6 box I run confluence, doesn't support mod_proxy_wstunnel out of the box. This will be my next try to get this working via SSL.

Sooner or later I will setup a CentOS 7 box with apache 2.4.x which brings already the module mod_proxy_wstunnel with it.

Cheers and good night
Andi

Ok, now it's getting a good night! This is what I did to make synchrony work through apache as ssl proxy for the websocket service.

 

# Check apache version (on my centos 6 box it's 2.2.15)
rpm -qa | grep httpd

# I would recommend to do all the build stuff in the next steps as a unprivileged user to not install or brake anything when doing this as root.
# ensure these packages are installed, otherwise commands below will fail:
# - svn
# - libtool

# Checkout apache source
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.15/ httpd-2.2.15

# Get patch and apply it
wget https://gist.github.com/vitkin/6661683/raw/873dd8b4de4ad1ff69757ffe48fc574374aedc57/apache-2.2-wstunnel.patch
cd httpd-2.2.15
patch -p1 -i ../apache-2.2-wstunnel.patch

# Build Apache
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.3.x srclib/apr-util
./buildconf
./configure --enable-proxy=shared --enable-proxy_wstunnel=shared
make

# Copy the module to apache installation
sudo cp modules/proxy/.libs/mod_proxy_wstunnel.so /etc/httpd/modules

# Create module load file
cd /etc/httpd/conf.d
echo "LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so" > mod_proxy_wstunnel.conf

# add proxy pass to the synchrony virtual host
<VirtualHost $external_IP4:8091 $external_IP6:8091>
    ServerName your.confluence.fqdn.org

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
    SSLCertificateKeyFile /path/to/key/file
    SSLCertificateFile /path/to/crt/file
    SSLCertificateChainFile /path/to/intermediate/if/needed
    SSLCACertificateFile /path/to/ca/file
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

    ProxyPass /synchrony/sockjs/v1/ ws://127.0.0.1:8091/synchrony/sockjs/v1/
    ProxyPass / http://127.0.0.1:8091/
    ProxyPassReverse / http://127.0.0.1:8091/
</VirtualHost>

# check configuration and reaload
apachectl -S
service httpd reload

 

Maybe it's useful to put this all together in a single post but that can be done after some sleep. smile

Cheers from http://www.sixhop.net/ side of life
Andi

Hello Andreas,

could you post whole apache config file for confluence virt host? We have the same problem with "wss:". Thanks

Regards,

Vojta

Hi Vojta,

it depends on your setup how your virtual host configuration need to look like. These is the VirtualHost I use for the synchrony proxy service:

<VirtualHost <IPv4>:8091 <IPv6>:8091>
    ServerAdmin <webmaster@example.com>
    DocumentRoot /var/www/html
    ServerName <confluence.example.com>

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
    SSLCertificateKeyFile </etc/ssl/server.key>
    SSLCertificateFile </etc/ssl/server.crt>
    SSLCertificateChainFile </etc/ssl/intermediate_ca.pem>
    SSLCACertificateFile </etc/ssl/ca.pem.cer>
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

    ProxyPass /synchrony/sockjs/v1/ ws://127.0.0.1:8091/synchrony/sockjs/v1/
    ProxyPass / http://127.0.0.1:8091/
    ProxyPassReverse / http://127.0.0.1:8091/

    ErrorLog logs/confluence-synchrony-error_log
    CustomLog logs/confluence-synchrony-access_log
</VirtualHost>

It can be that this virtual will work for you, but nobody can give a guarantee on that. All values in <> should be checked and adjusted to your needs starting by IPv4 and ending with the SSLCACertificateFile.

Would be great to hear if that works for you.

Cheers
Andi

Hi Andi,

thank you for response. I will try it. And what about virt confluence host :443? Do you use this - 

 

# Auth changes in 2.4 - see http://httpd.apache.org/docs/2.4/upgrading.html#run-time
	ProxyPass /synchrony http://&lt;internal_domain&gt;:8091/synchrony

	&lt;Location /synchrony&gt;
		Require all granted
		RewriteEngine on
		RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
		RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
		RewriteRule .* ws://&lt;internal_domain&gt;:8091%{REQUEST_URI} [P]
	&lt;/Location&gt;

 

 

how is mentioned on https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html

 

Regards,

Vojta

Hi - I'm running two concfluence instances with a path : my.domain.tld/confluenc1 & my.domain.tld/confluence2 . But the System is calling my.domain.tld/synchrony/... in both instances instead of my.domain.tld/confluence[1|2]/synchrony/ . Did anyone found a workaround how to configure it this way?

Hi,

https://confluence.atlassian.com/doc/configuring-synchrony-for-data-center-858772125.html

There is a config value named synchrony.context.path, looks like this is exactly what you're searching for. Have a try.

Cheers
Andi

Hey,

I made some changes to my configuration and it would be great if somebody with an apache 2.2 can verify that.

EDIT: build part of the module was missing in the explanation

Build mod_proxy_wstunnel for your apache version

# Check apache version (on my centos 6 box it's 2.2.15)
rpm -qa | grep httpd

# I would recommend to do all the build stuff in the next steps as an 
unprivileged user to not install or break anything when doing this as 
root.
# ensure these packages are installed, otherwise commands below will fail:
# - svn
# - libtool

# Checkout apache source
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.15/ httpd-2.2.15

# Get patch and apply it
wget https://gist.github.com/vitkin/6661683/raw/873dd8b4de4ad1ff69757ffe48fc574374aedc57/apache-2.2-wstunnel.patch
cd httpd-2.2.15
patch -p1 -i ../apache-2.2-wstunnel.patch

# Build Apache
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.3.x srclib/apr-util
./buildconf
./configure --enable-proxy=shared --enable-proxy_wstunnel=shared
make

# Copy the module to apache installation
sudo cp modules/proxy/.libs/mod_proxy_wstunnel.so /etc/httpd/modules

# Create module load file
cd /etc/httpd/conf.d
echo "LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so" &gt; mod_proxy_wstunnel.conf

 

Default proxy for the confluence application.

&lt;VirtualHost &lt;IPv4&gt;:443 &lt;IPv6&gt;:443&gt;
	...

	ProxyRequests Off
    ProxyVia Off
    SSLProxyEngine On

    ProxyPass / http://127.0.0.1:8080/
    ProxyPassReverse / http://127.0.0.1:8080/

	...
&lt;/VirtualHost&gt;

Connector in server.xml

&lt;Connector port="8080"
        maxHttpHeaderSize="8192"
        maxThreads="200"
        minSpareThreads="10"
        maxSpareThreads="75"
        enableLookups="false"
        redirectPort="8443"
        acceptCount="100"
        connectionTimeout="20000"
        debug="0"
        URIEncoding="UTF-8"
        proxyName="confluence.example.com"
        proxyPort="443"
        secure="true"
        scheme="https" /&gt;

Ensure to change the proxyName.

Add an additional VirtualHost for the synchrony service

&lt;VirtualHost &lt;IPv4&gt;:8091 &lt;IPv6&gt;:8091&gt;
	...
	SSLEngine on
	&lt;rest of ssl configuration&gt;
	...

    ProxyRequests Off
    ProxyVia Off
    SSLProxyEngine On

    ProxyPassMatch "^/synchrony/(.*/websocket.*$)" ws://127.0.0.1:8091/synchrony/$1
    ProxyPass /synchrony http://127.0.0.1:8091/synchrony

	...

&lt;/VirtualHost&gt;

Ensure only the external IPv4 and IPv6 addresses of your server are used here because confluence itself needs to be able to open the port 8091 on 127.0.0.1. The ProxyPassMatch statement is the trick. As long as confluence only calls websockets when there's a websocket in the url this will work.

Change synchrony.bind ip for your confluence instance

JAVA_OPTS="$JAVA_OPTS -Dsynchrony.bind=127.0.0.1"

Disable synchrony.proxy

The service is enabled by default, you can find this configuration statement in <confluence-home>/confluence.cfg.xml.

&lt;property name="synchrony.proxy.enabled"&gt;false&lt;/property&gt;

 

Restart Services

Depends on how you installed confluence, hopefully you know how to stop and start the application.

For apache you should do:

apachectl -S
service httpd restart

 

Would be great if somebody could check this explanation and reply if it's working this way for your setup as well.

Cheers
Andi

Hi,

 

I changed from Cloud to Server 6.0.1 yesterday, imported everything from the Cloud instance, have it up and running now BUT editor doesn't work. Same message as you have. I can't create Pages and also can not edit pages. 

Tried direct connection without SSL, no go.
Running NginX with SSL as proxy now, still, no go. 

And thats the reason I switched back to lovely 5.10 cheeky Anyway, look the inspection console to identify the error (could be different than ours?) which will give you clues of what is going on. Further more in the application folder/logs there is a new log file, something with synchrony.log. Take a look there also

As a workaround I turned off 'collaborative editing'. Let's see if future versions will get a fix for that smile

@Panos Karampis, how do you downgrade the Conflunece to 5.10?

If you did not made a backup before your upgrade, don't downgrade but simply turn off the collaborative editing in gneral settings -> collaborative editing.

Ohh.. wow.. thanks @Andreas Nitsche, it works for me smile

Same here, i'm not able to make the collaborative editing working with nginx. Hopefully i'm in a test env.

Which Version of nginx is in use for your confluence instance?

I had the same issue. after disabling collaborative editing now everything works again fine.

Just doing a clean install of confluence 6.0.2 using nginx and I have the same problem the editor does not come back I cannot create new pages

 

@Livio Sartori, you can simply turn off collaborative editing in general settings.

For you, (hopefully all of you) that still have problems upgrade to 6.0.2. According to their release notes there are some bug fixes regarding synchrony.

I am using NGINX, and I am also interested in the setting to make this work on NGINX.

 

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted Oct 24, 2018 in Confluence

Atlassian Research opportunity with Confluence templates

Do you use templates with Confluence? Take part in a remote 1-hr workshop. You'll receive USD $100 for your time!   We're looking for people to participate in a   remote 1-hr workshop...

1,089 views 17 14
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you