Confluence behind apache reverse proxy denies Collaborate

Question

Hi Community, we are running our Confluence behind an apache https proxy on a Debian9 Server. Confluence is the only app running on that machine. However Collaborate stops working and can not reach its backend. In the apache logs i can see following entries: [proxy:error] [pid 7851] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed Port 8091 is up and running, but somehow the websocketconnect fails miserably. Here is our Apache Config for the proxy: ApacheConfig SSL for Confluence #### confluence section TimeOut 1000 ServerName confluence.our.site DirectoryIndex index.html SSLEngine on #Certificates SSLCertificateFile /etc/apache2/ssl/our.site.crt SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem SSLCertificateKeyFile /etc/apache2/ssl/our.site.key SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem SSLProxyEngine on RewriteEngine on ProxyRequests off ProxyPreserveHost on RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" LogLevel info ssl:debug ErrorLog ${APACHE_LOG_DIR}/our.site-ssl-error.log CustomLog ${APACHE_LOG_DIR}/our.site-ssl-access.log combined Require all granted ProxyPass /synchrony http://localhost:8091/synchrony Require all granted RewriteEngine on RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P] ProxyPass / http://localhost:8090/ ProxyPassReverse / http://localhost:8090/ Options +FollowSymLinks +SymLinksIfOwnerMatch AllowOverride All Require all granted #Certbot ProxyPass /.well-known ! ProxyPassReverse /.well-known ! #additional ssl SSLProtocol all -SSLv2 -SSLv3 SSLHonorCipherOrder on SSLCompression off ###end of confluence section #### balsamiq rtc section Listen our.site:9083 SSLEngine on #Certificates SSLCertificateFile /etc/apache2/ssl/our.site.crt SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem SSLCertificateKeyFile /etc/apache2/ssl/our.site.key SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem ProxyRequests off ProxyPreserveHost On RewriteEngine on Require all granted ProxyPass / http://localhost:9083/ RewriteEngine on RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] RewriteRule .* ws://localhost:9083%{REQUEST_URI} [P] ###end of balsamiq rtc section We also have a port 80 vhost which redirects to the https: TimeOut 1000 UseCanonicalName On ServerAdmin root@localhost ServerName confluence.our.site DocumentRoot "/var/www/our.site" DirectoryIndex index.html ProxyRequests Off LogLevel info ErrorLog ${APACHE_LOG_DIR}/our.site-prod-error.log CustomLog ${APACHE_LOG_DIR}/our.site-access.log combined AllowOverride All Options +FollowSymLinks +SymLinksIfOwnerMatch Require all granted RewriteEngine on #RewriteCond %{THE_REQUEST} !/.well-known/acme-challenge/ [NC] RewriteCond %{SERVER_NAME} =confluence.solar-log.com RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent] our confluence server.xml: We are unable to open the ws sockets - this is also for balsamiq which also does not work. In order to use confluence we had to stop collaborate completely. But this is not what we wish. We need collaborate working. We have no running firewall on that server, but we have security tools rkhunter, aide, auditd, acct and lynis working. Can please someone help? Greetings, Tux

M_ Ludwig · Answer

Hi Alexis,

we have no iptables set up and no selinux running. SE is not even installed on the server.

The problem is also only located to Apache proxy/Synchrony both running on exactly this machine having problems to connect via localhost or 127.0.0.1.

Connecting to the synchrony port on localhost from a login shell of the server itself is working as a charm.

We checked everything twice and could not find a configuration different from our testsystem. This exactly is the reason why we are asking here - we simply have no idea where to search next.

Greetings,

Tux

Alexis Robert · Answer

Hi @M_ Ludwig ,

if you keep seeing the errors on your production servers, but the setup works fine on your test server, then the issue is definitely with your production server.

I would check again if there is not a SELinux or iptables policy preventing the connection to synchrony.

M_ Ludwig · Answer

Hi there,

we still have those severe problems which caused us to stop collaborate and thus balsamiq also is not working since it also uses websockets over synchrony.

Telnetting to the machines synchrony port works just fine from the local machine and also from a distant machine.

Also a wget of the heartbeat which works over synchrony is working and getting a file back with heartbeat ok.

But if we start Collaborate and thus restarting syncrony, we see in the apache ssl error logs always the following entries:

[proxy:error] [pid 17880] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:8091 (*) failed

[proxy_wstunnel:error] [pid 17880] [client 192.168.110.218:50896] AH02452: failed to make connection to backend: localhost

[proxy:error] [pid 19466] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed

[proxy_http:error] [pid 19466] [client 192.168.110.218:50892] AH01114: HTTP: failed to make connection to backend: localhost

For a test we installed confluence on another server (also Debian9 with same settings) with exactly the same configuration in confluence itself and apache - it works there like a charm. On our productive server we always see above nasty errors we can not find out where they are coming from.

Any help in that is appreciated.

Greetings,

Tux