Confluence Server behind HaProxy often slow or unresponsive

[bdegier]

Hi,

We used to run Confluence internally, it ran behind nginx as a reverse proxy which worked fine. I have recently made it available from over the internet and have used out existing HaProx 2.0.13 loadbalancers.

We are encountering some problems now. Often it won't save a changed page, it will sit there loading and then it shows an error that it currently cannot save the page. Sometimes it's also pretty slow opening the editor when clicking "Edit" on a page.

So sometimes when saving a page it just times out and sometimes it takes 5+ seconds for it to actually save and start loading the page again.

This is my HaProxy config (edited for privacy  reasons):

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend www
bind :443 ssl crt /etc/haproxy/certs/2020/REMOVED.crt
bind :80
option forwardfor

redirect scheme https code 301 if !{ ssl_fc }

acl app hdr(host) -i app.REMOVED.com
acl api hdr(host) -i api.REMOVED.com

use_backend app_nodes if app
use_backend api_nodes if api

# Confluence
acl docs hdr(host) -i docs.REMOVED.com
use_backend docs if docs

backend app_nodes
balance roundrobin
mode http
option http-server-close
option forwardfor
server app-ams7-lt-01 10.100.20.12:80 check
server app-ams7-lt-02 10.100.20.13:80 check

backend api_nodes
balance roundrobin
mode http
server api-ams7-lt-01 10.100.20.14:80 check
server api-ams7-lt-02 10.100.20.15:80 check

backend docs
balance roundrobin
mode http
option forwardfor
server docker-ams7-01 10.100.20.17:8092 check

listen stats
bind :9000 # Listen on localhost:9000
mode http
stats enable # Enable stats page
stats hide-version # Hide HAProxy version
stats realm Haproxy\ Statistics # Title text for popup window
stats uri /stats # Stats URI

 And this is my Docker-Compose file:

version: "2"
services:
confluence:
image: atlassian/confluence-server:latest
container_name: confluence
volumes:
- /opt/docker/containers/confluence/data:/var/atlassian/application-data/confluence
- /opt/docker/containers/confluence/mysql-connector-java-5.1.48.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/mysql-connector-java-5.1.48.jar
ports:
- '8092:8090'
- '8093:8091'
environment:
- JVM_MINIMUM_MEMORY=1024m
- JVM_MAXIMUM_MEMORY=4096m
- ATL_DB_TYPE=mysql
- ATL_JDBC_USER=REMOVED
- ATL_JDBC_PASSWORD=REMOVED
- ATL_JDBC_URL=jdbc:mysql://10.100.20.17:3306/confluence_docker
- ATL_PROXY_NAME=docs.REMOVED.com
- ATL_PROXY_PORT=443
- ATL_TOMCAT_SCHEME=https
- ATL_TOMCAT_SECURE=true
- TZ=Europe/Amsterdam
restart: unless-stopped

 I have no clue where to look anymore. When I used nginx I never had any issues. I used to have another backend in HaProxy for /synchrony but that caused issues with the editor not loading at all so I removed that. I did use that with nginx though.

[Ben Reiter]

We have the same problem. We run a couple instances via various HaProxy servers and they all show the same behavior:

90% of page interactions work fine without delay.

10% of Page Loads, Edits, Saves, Logins, Searches are very slow (5+ seconds), no timeouts though. I cannot identify any systematic pattern which would point to the cause.

We run a very large amount of services via Haproxy, we never had any such issue with any other application.

I will set up nginx for myself to see if the issues actually go away - If HAProxy is the cause, their developers are usually very good at helping out if useful diagnostic data can be provided.

Addendum: We use Haproxy 2.2.2 now, but the issue was present with 1.9.x already

[bdegier]

That's why I disabled Collaborative Editing, then it works fine with HaProxy.

[vitanetworks]

I moved from Confluence in a VM to containers too, very similar setup and performance tanked too even, not that it was performant to begin with. The hardware is more powerful and HAProxy has same config as before, only had to change the IP address.

The issue isn't the config, it's the software still running on Java. Containers aim to fix the same problem as Java's: allowing some code run on any platform. If containers already are their own thing, still delivering Java inside a container is lazy coding as it's only adding an extra layer of unnecessary bloat.

You might be able to improve things if you set higher timeouts in your HAProxy backends so it doesn't just discard the connection when it's taking too long and marks the server offline. Disabling health checks altogether and/or increasing keepalive time may help too.

For what it's worth, I decided to adding min/max memory values to Java after seeing you did--I skipped those. It's not gonna improve a thing but at least I'll have a fuller spec. Good luck! :)

[SmartLOXS]

This helped for me:

 environment:
- JVM_MINIMUM_MEMORY=1024m
- JVM_MAXIMUM_MEMORY=4096m

An yes, Java is often still slow and memory hungry. But our install is pretty snappy now I disabled collaborative editing in the settings. The whole Synchrony thing was holding things back, which is odd because behind nginx it worked like a charm.

[bdegier]

Anyone?