Confluence Server behind HaProxy often slow or unresponsive

Question

Hi,

We used to run Confluence internally, it ran behind nginx as a reverse proxy which worked fine. I have recently made it available from over the internet and have used out existing HaProx 2.0.13 loadbalancers.

We are encountering some problems now. Often it won't save a changed page, it will sit there loading and then it shows an error that it currently cannot save the page. Sometimes it's also pretty slow opening the editor when clicking "Edit" on a page.

So sometimes when saving a page it just times out and sometimes it takes 5+ seconds for it to actually save and start loading the page again.

This is my HaProxy config (edited for privacy  reasons):

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend www
bind :443 ssl crt /etc/haproxy/certs/2020/REMOVED.crt
bind :80
option forwardfor

redirect scheme https code 301 if !{ ssl_fc }

acl app hdr(host) -i app.REMOVED.com
acl api hdr(host) -i api.REMOVED.com

use_backend app_nodes if app
use_backend api_nodes if api

# Confluence
acl docs hdr(host) -i docs.REMOVED.com
use_backend docs if docs

backend app_nodes
balance roundrobin
mode http
option http-server-close
option forwardfor
server app-ams7-lt-01 10.100.20.12:80 check
server app-ams7-lt-02 10.100.20.13:80 check

backend api_nodes
balance roundrobin
mode http
server api-ams7-lt-01 10.100.20.14:80 check
server api-ams7-lt-02 10.100.20.15:80 check

backend docs
balance roundrobin
mode http
option forwardfor
server docker-ams7-01 10.100.20.17:8092 check

listen stats
bind :9000 # Listen on localhost:9000
mode http
stats enable # Enable stats page
stats hide-version # Hide HAProxy version
stats realm Haproxy\ Statistics # Title text for popup window
stats uri /stats # Stats URI

And this is my Docker-Compose file:

version: "2"
services:
confluence:
image: atlassian/confluence-server:latest
container_name: confluence
volumes:
- /opt/docker/containers/confluence/data:/var/atlassian/application-data/confluence
- /opt/docker/containers/confluence/mysql-connector-java-5.1.48.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/mysql-connector-java-5.1.48.jar
ports:
- '8092:8090'
- '8093:8091'
environment:
- JVM_MINIMUM_MEMORY=1024m
- JVM_MAXIMUM_MEMORY=4096m
- ATL_DB_TYPE=mysql
- ATL_JDBC_USER=REMOVED
- ATL_JDBC_PASSWORD=REMOVED
- ATL_JDBC_URL=jdbc:mysql://10.100.20.17:3306/confluence_docker
- ATL_PROXY_NAME=docs.REMOVED.com
- ATL_PROXY_PORT=443
- ATL_TOMCAT_SCHEME=https
- ATL_TOMCAT_SECURE=true
- TZ=Europe/Amsterdam
restart: unless-stopped

I have no clue where to look anymore. When I used nginx I never had any issues. I used to have another backend in HaProxy for /synchrony but that caused issues with the editor not loading at all so I removed that. I did use that with nginx though.

bdegier · Answer

That's why I disabled Collaborative Editing, then it works fine with HaProxy.

Ben Reiter · Answer

We have the same problem. We run a couple instances via various HaProxy servers and they all show the same behavior:

90% of page interactions work fine without delay.

10% of Page Loads, Edits, Saves, Logins, Searches are very slow (5+ seconds), no timeouts though. I cannot identify any systematic pattern which would point to the cause.

We run a very large amount of services via Haproxy, we never had any such issue with any other application.

I will set up nginx for myself to see if the issues actually go away - If HAProxy is the cause, their developers are usually very good at helping out if useful diagnostic data can be provided.

Addendum: We use Haproxy 2.2.2 now, but the issue was present with 1.9.x already

SmartLOXS · Answer

This helped for me:

environment:
- JVM_MINIMUM_MEMORY=1024m
- JVM_MAXIMUM_MEMORY=4096m

An yes, Java is often still slow and memory hungry. But our install is pretty snappy now I disabled collaborative editing in the settings. The whole Synchrony thing was holding things back, which is odd because behind nginx it worked like a charm.