nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Direct synchrony access behind reverse proxy not working

Robin Jansohn

I'm trying to migrate our Confluence Server installation to run in a docker container and behind a reverse proxy (nginx).

I have a working setup but only if I do not try to configure nginx to directly contact synchrony.

nginx

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091/synchrony;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}

nginx logs

172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony/heartbeat?_=1610721370126 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony-proxy/heartbeat?_=1610721370127 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"

atlassian-synchrony.log

2021-01-15 15:36:11,135 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,135 INFO [async-dispatch-1] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
2021-01-15 15:36:11,221 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,220 INFO [async-dispatch-3] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}

404 response payload

{"message":"Not Found"}

As soon as I remove the "location /synchrony" block above it works (via synchrony-proxy).

Can anybody share some light what I'm doing wrong?

Accepted answers

Thiago Masutti

@Robin Jansohn

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional resolve and variables configuration, this would be the nginx conf for Synchrony:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://10.10.0.3:8091/synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   set $upstream_synchrony confluence-test:8091;
   proxy_pass http://$upstream_synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

The important portion is on the proxy_pass configuration where we don't need to add the /synchrony path.

Kind regards,
Thiago Masutti

All comments

Dave Theodore [Coyote Creek Consulting]

What version of Confluence? Modern versions no longer need you to handle the Synchrony proxy. It's done natively in the Tomcat configuration.

Robin Jansohn

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case IMO as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

Robin Jansohn

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

Thiago Masutti

Hi @Robin Jansohn

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

Regards,
Thiago Masutti

Thiago Masutti

@Robin Jansohn

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional resolve and variables configuration, this would be the nginx conf for Synchrony:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://10.10.0.3:8091/synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   set $upstream_synchrony confluence-test:8091;
   proxy_pass http://$upstream_synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

The important portion is on the proxy_pass configuration where we don't need to add the /synchrony path.

Kind regards,
Thiago Masutti

Robin Jansohn

@Thiago Masutti

thanks again for helping me track down the problem even if it wasn't an Atlassian problem after all!

The reason why it didn't work when using the resolver is explained in the official docs (https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass):

When variables are used in proxy_pass:
location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}
In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

While not super important it is still better to specify location /synchrony/ because then requests to synchrony-proxy are not sent down the wrong route to synchrony. That is by the way the reason why it did not work at all in my original configuration and why I even got a 404 when sending the synchrony-proxy calls.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

server

confluence-server

nginx

synchrony