How to configure Apache for Confluence 6.2 Synchrony cluster

Hi

I am wondering if someone has managed to configure Apache for Synchrony Cluster proxy in?

I have well established configuration for Jira and Confluence DC cluster based on Apache 2.4 + MOD_JK + AjpNio connector.

Now I would like to add Synchrony in it but I can't find out any sample on Atlassian for that.

There is only one reference to single server option.

Regards,
Areg

6 answers

1 accepted

Hi All

With help of my collegaues, PS Yilin Mo and TAM Kevin we have solved this puzzle.

 

So the steps are involving:


#===================== Worker Section Start =====================
# Confluence Load Balancer Worker
worker.list=confluence-lb
worker.confluence-lb.type=lb
worker.confluence-lb.balance_workers=confluence-n1,confluence-n2
worker.confluence-lb.sticky_session=true
worker.confluence-lb.session_cookie=JSESSIONID
# Worker for Confluence Node 1 - HTTPS
worker.confluence-n1.type=ajp13
worker.confluence-n1.ping_mode=I
worker.confluence-n1.socket_connect_timeout=5000
worker.confluence-n1.host=confluence-node1
worker.confluence-n1.port=8600
worker.confluence-n1.route=confluence-node1
# Worker for Confluence Node 2 - HTTPS
worker.confluence-n2.type=ajp13
worker.confluence-n2.ping_mode=I
worker.confluence-n2.socket_connect_timeout=5000
worker.confluence-n2.host=confluence-node2
worker.confluence-n2.port=8600
worker.confluence-n2.route=confluence-node2
#=====================  Worker Section End  =====================
  • Configure Confluence to accept connections from MOD_JK
        <!-- Main Connector Port -->
        <Connector port="8600"

            maxThreads="250"
            minSpareThreads="25"
            connectionTimeout="20000"

            enableLookups="false"
            protocol="org.apache.coyote.ajp.AjpNioProtocol"
            redirectPort="8443"
            acceptCount="10"
            debug="0"
            URIEncoding="UTF-8"

            proxyName="confluence.something.com"
            proxyPort="443"

        />
  • Configure Apache for MOD_JK
    JkMount     /*                 confluence-lb
    JkUnMount   /html/*            confluence-lb
    JkUnMount   /synchrony/*       confluence-lb
    JkUnMount   /balancer-manager  confluence-lb
    <Location /balancer-manager>
        SetHandler balancer-manager

        Require                <access list>
    </Location>

    #Synchrony configuration
    ProxyRequests Off
    ProxyPreserveHost On

    # Proxy Balancer
    <Proxy balancer://confluence-synchrony-http>
        # Synchrony Node 1
        BalancerMember        http://synchrony-node1:9100 route=confluence-synnode1
        # Synchrony node 2
        BalancerMember        http://synchrony-node2:9100 route=confluence-synnode2

        # Security
        Require                all granted

        # Load Balancer Settings
        # We are not really balancing anything in this setup, but need to configure this
        ProxySet            lbmethod=byrequests
        ProxySet            stickysession=JSESSIONID
    </Proxy>
    <Proxy balancer://confluence-synchrony-ws>
        # Synchrony Node 1
        BalancerMember        ws://synchrony-node1:9100 route=confluence-synnode1
        # Synchrony Node 2
        BalancerMember        ws://synchrony-node2:9100 route=confluence-synnode1

        # Security
        Require                all granted

        # Load Balancer Settings
        # We are not really balancing anything in this setup, but need to configure this
        ProxySet            lbmethod=byrequests
        ProxySet            stickysession=JSESSIONID
    </Proxy>

    # https://serverfault.com/questions/616370/configuring-apache-2-4-mod-proxy-wstunnel-for-socket-io-1-0x
    RewriteEngine     On
    RewriteCond       %{REQUEST_URI}       ^/synchrony/(.*)/websocket            [NC]
    RewriteRule       /(.*)                balancer://confluence-synchrony-ws/$1  [P,L]
RewriteCond %{REQUEST_URI}       ^/synchrony/(.*)/bayeux-sync1                           [NC]
RewriteRule       /(.*)                balancer://confluence-synchrony-pro0-ws/$1      [P,L]

    #ProxyPass        /balancer-manager    !
    ProxyPass         /synchrony            balancer://confluence-synchrony-http/synchrony
    ProxyPassReverse  /synchrony            balancer://confluence-synchrony-http/synchrony

 

Hopefully this is all.

i created a DNS entry for the VIP on the load balancer in front of the two synchrony nodes.  I changed my mod proxy vhost file to call the vip name.

e.g.

 

 

ProxyPass /synchrony https://synchrony.xxx.com/synchrony

    ProxyPassReverse /synchrony https://synchrony.xxx.com/synchrony

    <Location /synchrony>

        Require all granted

        RewriteEngine on

        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

        RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]

        RewriteRule .* ws://synchrony.xxx.com%{REQUEST_URI} [P]

    </Location>

 

because i made the call over SSL i also had to add this to the vhost file.

 

    SSLProxyCheckPeerCN off

    SSLProxyCheckPeerName off

 

Hi Sean,

 

How you accomplished ws load balancing.?

 

I have a F5 in front of two synchrony nodes it is load balancing http/s traffics but not ws .Do we have to enable some configuration in F5?

I had to drop the SSL connection.  =(  I call the vip as 8091 and pass through now.  

Thanks.

 

Will that mean you put a rewwrite rule like this.

 RewriteRule .* ws://synchrony.xxx.com:8091%{REQUEST_URI} [P]


1.Is Ws traffic passing through F5 with this config?
2.Do we have to do any settings in F5 for this?

That will be fine but my apache load balancing is configured only for synchrony servers as I have F5 for confluence load balancing.I had discussion with PS on this session stickiness part for synchrony.I guess for internal requests to synchrony JSESSION id will not be passed.So not sure apache sticky session with JSESSION ID makes sense for synchrony hub.

Will post my observation once I get a concrete evidence.

As per vendor update LB for synchrony doesn't need session stickiness.

0 votes
Ann Worley Atlassian Team May 24, 2017

In front of your Data Center cluster there should be a load balancer. Configure Apache in front of the IP being used for the cluster, using the same Apache settings you would use for the single server.

This doc has a sample config in an expand macro just under the text "If you're using Confluence 6.0 or later with Synchrony (which is required for collaborative editing), you'll need to use Apache 2.4.10 or later..."

Hi Ann

That document doesn't describe how to do that for multiple synchrony instances running on a multiple nodes.

 

I know how to configure MOD_JK with Sticky Session for 2 node Confluence DC and it is working quite good.

Something like this https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html but again it is for single host. How about multiple hosts?

 

Now when I have started to prepare for 6.2 update I have a feeling that there is a lack of documentation around it.

 

Regards,

Areg

Right

For the Datacenter Jira or Confluence you can do following:

* with mod_proxy and mod_ajp

 # Proxy Balancer
 <Proxy balancer://confluence-cluster>
  # Confluence Node 1
  BalancerMember ajp://confluence-node1:8600 route=node1 timeout=1800
  # Confluence Node 2
  #BalancerMember ajp://confluence-node1:8600 route=node2 timeout=1800

  # Security
  Order  Deny,Allow
  #Deny from None
  Allow from All

  # Load Balancer Settings
  # We are not really balancing anything in this setup, but need to configure this
  ProxySet lbmethod=byrequests
  ProxySet stickysession=JSESSIONID
 </Proxy>

 ProxyRequests  Off
 ProxyPreserveHost On

 ProxyPass  /  balancer://confluence-cluster/
 ProxyPassReverse /  balancer://confluence-cluster/

* with mod_jk

"workers.properties" file

 #===================== Worker Section Start =====================
 # Load Balancer Worker
 worker.list=confluence-lb
 worker.confluence-lb.type=lb
 worker.confluence-lb.balance_workers=confluence-node1,confluence-node2
 worker.confluence-lb.sticky_session=true
 worker.confluence-lb.session_cookie=JSESSIONID
 # Worker for Node 1 - HTTPS
 worker.confluence-node1.type=ajp13
 worker.confluence-node1.ping_mode=I
 worker.confluence-node1.socket_connect_timeout=5000
 worker.confluence-node1.host=confluence-node1
 worker.confluence-node1.port=8600
 worker.confluence-node1.route=confluence-node1
 # Worker for Node 2 - HTTPS
 worker.confluence-node2.type=ajp13
 worker.confluence-node2.ping_mode=I
 worker.confluence-node2.socket_connect_timeout=5000
 worker.confluence-node2.host=confluence-node2
 worker.confluence-node2.port=8600
 worker.confluence-node2.route=confluence-node2
 #=====================  Worker Section End  =====================

Apache section

 JkMount    /*     confluence-lb

For Synchrony I can only see example in https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html I can only see section 6 for Synchrony like

6. Configure your load balancer for Synchrony.
Your load balancer must support WebSockets (for example NGINX 1.3 or later, Apache httpd 2.4, IIS 8.0 or later) and session affinity. 
SSL connections must be terminated at your load balancer so that Synchrony can accept XHR requests from the web browser. 

and nothing

Then in https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288

# Put this after the other LoadModule directives
LoadModule jk_module modules/mod_jk.so

# Adding mod_proxy w/ wstunnel and mod_rewrite for Synchrony support
LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module /usr/lib/apache2/modules/mod_proxy_wstunnel.so
LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

# Put this in the main section of your configuration (or virtual host, if using Apache virtual hosts)
JkWorkersFile conf/workers.properties
JkLogFile logs/apache2/mod_jk.log
JkLogLevel info

<VirtualHost *:80>
 #Synchrony configuration
 ProxyRequests Off
 ProxyPreserveHost On

 <Proxy *>
     Require all granted
 </Proxy>

 ProxyPass /synchrony http://localhost:8091/synchrony
 <Location /synchrony>
  Require all granted
  RewriteEngine on
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P]
 </Location>
 
 # mod_jk configuration
 JkMount /confluence worker1
 JkMount /confluence/* worker1

</VirtualHost>

I have also asked PS and TAM and looks like there is no exact example of 2 node and more Synchrony configuration on Apache or else even the section 5 in https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html tells to start process on each node.

Small section in https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288 tells how to do the WebSockets redirection:

 ProxyPass /synchrony http://localhost:8091/synchrony
 <Location /synchrony>
  Require all granted
  RewriteEngine on
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P]
 </Location>

But not how to load balance between the Synchrony nodes.

 

Regards,

Areg

Ann Worley Atlassian Team May 30, 2017

As you mentioned, you have a ticket open with Atlassian's Premier Support. It looks like they are still investigating. When you get a configuration that works, I hope you will come back to the Community and share an example.

Slight hijack... and apologies for that but very related.

I have synchrony behind it's pown VIP and just call it as 8091 and all that works well.  My issue is for the app side.  When I have more than 1 server on the VIP for the app side active, users could not check boxes or change status macros with out being in edit mode.  I was able to fix this with changing to use the JSESSIONID cookie in the F5 in front of the app.  It appears though that the Apache/mod_proxy server in front of the app tier is breaking the JSESSIONID cookie.  For the life of me I can't figure out why / how.  I am about to switch to NGiNX instead.

HI Areg,

 

I tried configuring the same in  apache for synchrony hub.

Also i noticed that the synchrony traffic is wss:xxxx/synchrony

 

So i used the configuration for rewrite in apache

 

<Location /synchrony>

    Require all granted

   RewriteEngine on

    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]

   RewriteRule /(.*) balancer://confluence-synchrony-ws/$1  [P,L]

</Location>

 

 

But I am getting response code of 204 no content even though connection is getting success.

Hi

Which version of Confluence you are running?

In latest versions it has changed.

You can check it by using Firefox network monitor.

I will post fixed configuration once will be online.

What changed and in which version???


In new version of Synchrony it is looking ^/synchrony/(.*)/bayeux-sync1 URL instead of ^/synchrony/(.*)/websocket and it is not documented anywhere ...

It is possible to detect by usung Firefox or Chrome Network Monitor and see which request fails on Websocket and add rewrite condition for it.

I have updated my answer for Apache - working as a charm for us.

Hi Areg,

Thanks for sharing this .I configured the apache to act as lb for synchrony hub.

Configured jsession id for session stickiness.Is there any way to confirm this is working or not?

Use developer tools in chrome 

application tab

expand the cookies and watch if the jsessionid cookie stays the same when running more than 1 app server.

Hi All,

We are observing below behavior.

 

1.intermittently in one node suddenly saving will fail and it will fix itself after 5 min.

2.Browser will give 204 response code.

3.Logs are not giving much info.

 

Does anyone faced this issue?.We are also using apache for loadbalancing synchrony cluster.

Suggest an answer

Log in or Sign up to answer
Community showcase
Posted Oct 11, 2018 in Confluence

What are your project planning tips?

Hello Community,  Jessica here from the Confluence product marketing team! Today I wanted to get your takes on project planning –– what works, what doesn’t, how do you know if you’re doing it r...

258 views 1 4
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you