Direct synchrony access behind reverse proxy not working

[Robin Jansohn]

I'm trying to migrate our Confluence Server installation to run in a docker container and behind a reverse proxy (nginx).

I have a working setup but only if I do not try to configure nginx to directly contact synchrony.

nginx

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091/synchrony;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}

nginx logs

172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony/heartbeat?_=1610721370126 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony-proxy/heartbeat?_=1610721370127 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"

atlassian-synchrony.log

2021-01-15 15:36:11,135 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,135 INFO [async-dispatch-1] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
2021-01-15 15:36:11,221 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,220 INFO [async-dispatch-3] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}

404 response payload

{"message":"Not Found"}

As soon as I remove the  "location /synchrony" block above it works (via synchrony-proxy).

Can anybody share some light what I'm doing wrong?

[Thiago Masutti · Accepted solution]

@Robin Jansohn 

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional resolve and variables configuration, this would be the nginx conf for Synchrony:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://10.10.0.3:8091/synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   set $upstream_synchrony confluence-test:8091;
   proxy_pass http://$upstream_synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

The important portion is on the proxy_pass configuration where we don't need to add the /synchrony path.

Kind regards,
Thiago Masutti

[Robin Jansohn]

@Thiago Masutti 

thanks again for helping me track down the problem even if it wasn't an Atlassian problem after all!

The reason why it didn't work when using the resolver is explained in the official docs (https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass):

When variables are used in proxy_pass:

location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}

In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

 While not super important it is still better to specify location /synchrony/ because then requests to synchrony-proxy are not sent down the wrong route to synchrony. That is by the way the reason why it did not work at all in my original configuration and why I even got a 404 when sending the synchrony-proxy calls.

[Dave Theodore [Coyote Creek Consulting]]

What version of Confluence? Modern versions no longer need you to handle the Synchrony proxy. It's done natively in the Tomcat configuration.

[Robin Jansohn]

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case IMO as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

[Robin Jansohn]

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

[Thiago Masutti]

Hi @Robin Jansohn 

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

Regards,
Thiago Masutti

[Robin Jansohn]

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

That is exactly the scenario which works as I wrote in the opening post.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

This is what I'm trying to achieve and it fails.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

I haven't changed the default port, so 8091. But I do not expose it outside of the container. It is only reachable within the attached docker network so nginx and confluence container can reach each other. This way it is impossible to reach synchrony directly for an external client without passing through nginx.

[Thiago Masutti]

Hey @Robin Jansohn 

Thanks for the feedback.

Would you be willing to go on a screen share and work together on this?
If so, this URL allows you to book some time in my agenda based on your best availability.
When I receive the booking, I will update the invite with a Zoom call.

Note that this is a temporary URL and it will expire in 2 hours. :)

Regards,
Thiago Masutti

[Robin Jansohn]

Hi @Thiago Masutti 

I just packed up work yesterday after writing that post. We can do a screen share today if you have time.

Kind regards,
Robin Jansohn

[Thiago Masutti]

Hi @Robin Jansohn 

That's fine. It seems we are on different timezones.

This is a link to Zoom call.

I will keep it running for the next hours, so feel free to join it so we can take a look at it together.

Regards,
Thiago Masutti

[Robin Jansohn]

I don't think I'm allowed to use Zoom in my company. Any chance we can do a Microsoft Teams call?

[Thiago Masutti]

Sure.

Send me the MS Teams link and I will join

[Robin Jansohn]

EDIT: removed link

[Robin Jansohn]

@Thiago Masutti 

I got it working now with the docker DNS resolver. Here is my nginx configuration:

client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony/ {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}

[Thiago Masutti]

@Robin Jansohn 

I'm almost sure that using location /synchrony/ makes nginx to send all the requests to the default / location, which will then use the Tomcat Synchrony Proxy.

You can confirm by looking at the developer tools and validate requests are sent to /synchrony and not to /synchrony-proxy.

I was able to reproduce the issue if I'm using the resolver directive in nginx, so I'm testing things here.

Regards,
Thiago