Collaborative editing doesn't work with NGINX reverse proxy

[Oliver Schmidt]

Hi everyone,

it seems that I have a problem with confluence behind an NGINX SSL proxy and collaborative editing. 

Follwing infrastructure was build:

NGINX SSL reverse proxy (OpenBSD)

Configuration like in: https://confluence.atlassian.com/confeap/running-confluence-behind-nginx-with-ssl-849150880.html

• ssl enabled
• gzip off
• OCSP stapling on

# Header
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1;mode=block";
add_header Strict-Transport-Security "max-age=31526000;includeSubdomains;preload";
add_header Content-Security-Policy "script-src 'self'";
add_header X-Content-Type-Options "nosniff" always;
add_header Public-Key-Pins 'pin-sha256="xxx"; pin-sha256="xxx"; max-age=23328000; includeSubDomains';

[...]

location / {
  client_max_body_size 100m;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_pass http://10.10.10.10:8090;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-Proto $scheme;
}
location /synchrony {
  client_max_body_size 100m;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_pass http://10.10.10.10:8091/synchrony;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";

}

Confluence application server (Ubuntu 16.04.2)

Configuration like in: https://confluence.atlassian.com/doc/installing-confluence-on-linux-143556824.html

• PostgreSQL running under localhost:5432

/opt/atlassian/confluence/conf/server.xml:

<Server port="8000" shutdown="SHUTDOWN" debug="0">
<Service name="Tomcat-Standalone">
<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" minSpareThreads="10"
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
proxyName="confluence.xxx.xx" proxyPort="443" scheme="https" />

Symptons:

1. After logging into confluence I can see the first HTTP 404 errors in NGINX access.log while I am asking myself if /undefined/rest... could be correct:

x.x.x.x - - [07/Jun/2017:14:36:59 +0200] "GET /undefined/rest/supportHealthCheck/1.0/check/admin HTTP/1.1" 404 31798 "https://confluence.xxx.xx/welcome.action" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" "-"
x.x.x.x - - [07/Jun/2017:14:37:00 +0200] "POST /undefined/rest/analytics/1.0/publish/bulk HTTP/1.1" 404 31792 "https://confluence.xxx.xx/welcome.action" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" "-"

2. After a click on Create ... on top bar, I get this behavior: https://confluence.atlassian.com/confkb/editor-fails-to-load-in-confluence-6-x-due-to-could-not-initialize-class-org-xerial-snappy-snappy-error-859462192.html but not having the error messages in the log files which are described in this document.

I also get more HTTP 404 error on NGINX access.log like:

x.x.x.x - - [07/Jun/2017:14:46:49 +0200] "POST /pages/undefined/json/startheartbeatactivity.action HTTP/1.1" 404 31865 "https://confluence.xxx.xx/pages/createpage.action?spaceKey=ds" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" "-"

and a lot of:

x.x.x.x - - [07/Jun/2017:14:47:49 +0200] "POST /pages/undefined/json/startheartbeatactivity.action HTTP/1.1" 404 31865 "https://confluence.xxx.xx/pages/resumedraft.action?draftId=851969&draftShareId=8f486ff4-b7e3-4e57-a036-f712050868a0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" "-"

3. A click under Settings -> General configuration -> Collaborative editing produces a lot of WEBUI errors

and more 404 errors in NGINX access.log:

x.x.x.x - - [07/Jun/2017:14:54:17 +0200] "GET /admin/confluence-collaborative-editor-plugin/undefined/rest/synchrony-interop/synchrony-status HTTP/1.1" 404 31792 "https://confluence.xxx.xx/admin/confluence-collaborative-editor-plugin/configure.action" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" "-"

Resolution tries:

• Check if 8090 and 8091 are accessible from the NGINX SSL reverse proxy
• Resolution suggestions written in https://confluence.atlassian.com/confkb/editor-fails-to-load-in-confluence-6-x-due-to-could-not-initialize-class-org-xerial-snappy-snappy-error-859462192.html: But /tmp is not flagged with noexec.
• Nearly every suggestion inside: https://confluence.atlassian.com/doc/troubleshooting-collaborative-editing-858772087.html

Conclusion:

After reviewing this text I think the main problem seems to be the "undefined" but I have no clue where this comes from. I would be thankful for every help and hint I could get.

Additional question:

Is it normal that there are a lot of ESTABLISHED localhost connections from confluence to the PostgreSQL database (around 21) and a lot of TIME_WAIT (around 15) even if there is no ongoing traffic on the website?

[AnnWorley]

Thank you for the detailed description. Your NGINX configuration looks the same as in Running Confluence behind NGINX with SSL.

The behavior in the screen shots is consistent with some plugins being disabled. Please check the following ones:

• Confluence Collaborative Editor Plugin (should be enabled but it is good to confirm)
• Synchrony Interop Bootstrap Plugin (should be disabled based on the behavior)

Scheduled jobs like indexing can take up database connections. You may see more about the activity by running this SQL on your database - please let us know if you find anything concerning:

select *
from pg_stat_activity
where datname = 'db_name';