Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Next challenges

Recent achievements

  • Global
  • Personal

Recognition

  • Give kudos
  • Received
  • Given

Leaderboard

  • Global

Trophy case

Kudos (beta program)

Kudos logo

You've been invited into the Kudos (beta program) private group. Chat with others in the program, or give feedback to Atlassian.

View group

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Direct synchrony access behind reverse proxy not working

I'm trying to migrate our Confluence Server installation to run in a docker container and behind a reverse proxy (nginx).

I have a working setup but only if I do not try to configure nginx to directly contact synchrony.

nginx

server {
listen 443 ssl;
server_name confluencetest.mycompany.com;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

resolver 127.0.0.11 valid=30s;
set $upstream_confluence confluence;
proxy_pass http://$upstream_confluence:8090;
}

location /synchrony {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

resolver 127.0.0.11 valid=30s;
set $upstream_confluence confluence;
proxy_pass http://$upstream_confluence:8091/synchrony;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}

nginx logs

172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony/heartbeat?_=1610721370126 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony-proxy/heartbeat?_=1610721370127 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"

atlassian-synchrony.log

2021-01-15 15:36:11,135 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,135 INFO [async-dispatch-1] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
2021-01-15 15:36:11,221 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,220 INFO [async-dispatch-3] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}

404 response payload

{"message":"Not Found"}

As soon as I remove the  "location /synchrony" block above it works (via synchrony-proxy).

Can anybody share some light what I'm doing wrong?

2 answers

1 accepted

0 votes
Answer accepted

@Robin Jansohn 

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional resolve and variables configuration, this would be the nginx conf for Synchrony:

location /synchrony {
client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://10.10.0.3:8091/synchrony;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}

 

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

location /synchrony {
client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
set $upstream_synchrony confluence-test:8091;
proxy_pass http://$upstream_synchrony;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}

 

The important portion is on the proxy_pass configuration where we don't need to add the /synchrony path.

 

Kind regards,
Thiago Masutti

@Thiago Masutti 

thanks again for helping me track down the problem even if it wasn't an Atlassian problem after all!

The reason why it didn't work when using the resolver is explained in the official docs (https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass):

When variables are used in proxy_pass:

location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}

In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

 While not super important it is still better to specify location /synchrony/ because then requests to synchrony-proxy are not sent down the wrong route to synchrony. That is by the way the reason why it did not work at all in my original configuration and why I even got a 404 when sending the synchrony-proxy calls.

What version of Confluence? Modern versions no longer need you to handle the Synchrony proxy. It's done natively in the Tomcat configuration.

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case IMO as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts

Hi @Robin Jansohn 

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

Regards,
Thiago Masutti

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

That is exactly the scenario which works as I wrote in the opening post.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

This is what I'm trying to achieve and it fails.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

I haven't changed the default port, so 8091. But I do not expose it outside of the container. It is only reachable within the attached docker network so nginx and confluence container can reach each other. This way it is impossible to reach synchrony directly for an external client without passing through nginx.

Hey @Robin Jansohn 

Thanks for the feedback.

Would you be willing to go on a screen share and work together on this?
If so, this URL allows you to book some time in my agenda based on your best availability.
When I receive the booking, I will update the invite with a Zoom call.

Note that this is a temporary URL and it will expire in 2 hours. :)

Regards,
Thiago Masutti

Hi @Thiago Masutti 

I just packed up work yesterday after writing that post. We can do a screen share today if you have time.

Kind regards,
Robin Jansohn

Hi @Robin Jansohn 

That's fine. It seems we are on different timezones.

This is a link to Zoom call.

I will keep it running for the next hours, so feel free to join it so we can take a look at it together.

Regards,
Thiago Masutti

I don't think I'm allowed to use Zoom in my company. Any chance we can do a Microsoft Teams call?

Sure.

Send me the MS Teams link and I will join

@Thiago Masutti 

I got it working now with the docker DNS resolver. Here is my nginx configuration:

client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony/ {
resolver 127.0.0.11 valid=30s;
set $upstream_confluence confluence;
proxy_pass http://$upstream_confluence:8091;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}

@Robin Jansohn 

 

I'm almost sure that using location /synchrony/ makes nginx to send all the requests to the default / location, which will then use the Tomcat Synchrony Proxy.

You can confirm by looking at the developer tools and validate requests are sent to /synchrony and not to /synchrony-proxy.

 

I was able to reproduce the issue if I'm using the resolver directive in nginx, so I'm testing things here.

Regards,
Thiago

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
SERVER
VERSION
7.4.6
TAGS
Community showcase
Published in Confluence Cloud

Get to know the Confluence team!

Go “behind the screen” to meet some of the Confluence Cloud team. In this video series, we tackle some of the hard-hitting questions you never knew you wanted the answer to!  Meet some of the ...

236 views 0 10
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you