Heads up! On March 5, starting at 4:30 PM Central Time, our community will be undergoing scheduled maintenance for a few hours. During this time, you might find the site temporarily inaccessible. Thanks for your patience. Read more.

×
Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Product Q&A
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

Direct synchrony access behind reverse proxy not working

Robin Jansohn
Robin Jansohn January 15, 2021

I'm trying to migrate our Confluence Server installation to run in a docker container and behind a reverse proxy (nginx).

I have a working setup but only if I do not try to configure nginx to directly contact synchrony.

nginx

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091/synchrony;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}

nginx logs

172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony/heartbeat?_=1610721370126 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony-proxy/heartbeat?_=1610721370127 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"

atlassian-synchrony.log

2021-01-15 15:36:11,135 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,135 INFO [async-dispatch-1] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
2021-01-15 15:36:11,221 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,220 INFO [async-dispatch-3] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}

404 response payload

{"message":"Not Found"}

As soon as I remove the  "location /synchrony" block above it works (via synchrony-proxy).

Can anybody share some light what I'm doing wrong?

Answer
Watch
Like Be the first to like this
1695 views

2 answers

1 accepted

1 vote
Answer accepted
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 20, 2021

@Robin Jansohn 

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional resolve and variables configuration, this would be the nginx conf for Synchrony:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://10.10.0.3:8091/synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

 

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   set $upstream_synchrony confluence-test:8091;
   proxy_pass http://$upstream_synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}

 

The important portion is on the proxy_pass configuration where we don't need to add the /synchrony path.

 

Kind regards,
Thiago Masutti

View More Comments
Robin Jansohn
Robin Jansohn January 20, 2021

@Thiago Masutti 

thanks again for helping me track down the problem even if it wasn't an Atlassian problem after all!

The reason why it didn't work when using the resolver is explained in the official docs (https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass):

When variables are used in proxy_pass:

location /name/ {
    proxy_pass http://127.0.0.1$request_uri;
}

In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI.

 While not super important it is still better to specify location /synchrony/ because then requests to synchrony-proxy are not sent down the wrong route to synchrony. That is by the way the reason why it did not work at all in my original configuration and why I even got a 404 when sending the synchrony-proxy calls.

Like
Reply
0 votes
Dave Theodore [Coyote Creek Consulting]
Dave Theodore [Coyote Creek Consulting]
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
Become a leader
January 18, 2021

What version of Confluence? Modern versions no longer need you to handle the Synchrony proxy. It's done natively in the Tomcat configuration.

View More Comments
Robin Jansohn
Robin Jansohn January 18, 2021

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case IMO as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
Like
Robin Jansohn
Robin Jansohn January 18, 2021

We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
Like
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 19, 2021

Hi @Robin Jansohn 

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

Regards,
Thiago Masutti

Like
Robin Jansohn
Robin Jansohn January 19, 2021

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

That is exactly the scenario which works as I wrote in the opening post.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

This is what I'm trying to achieve and it fails.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

I haven't changed the default port, so 8091. But I do not expose it outside of the container. It is only reachable within the attached docker network so nginx and confluence container can reach each other. This way it is impossible to reach synchrony directly for an external client without passing through nginx.

Like
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 19, 2021

Hey @Robin Jansohn 

Thanks for the feedback.

Would you be willing to go on a screen share and work together on this?
If so, this URL allows you to book some time in my agenda based on your best availability.
When I receive the booking, I will update the invite with a Zoom call.

Note that this is a temporary URL and it will expire in 2 hours. :)

Regards,
Thiago Masutti

Like
Robin Jansohn
Robin Jansohn January 20, 2021

Hi @Thiago Masutti 

I just packed up work yesterday after writing that post. We can do a screen share today if you have time.

Kind regards,
Robin Jansohn

Like
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 20, 2021

Hi @Robin Jansohn 

That's fine. It seems we are on different timezones.

This is a link to Zoom call.

I will keep it running for the next hours, so feel free to join it so we can take a look at it together.

Regards,
Thiago Masutti

Like
Robin Jansohn
Robin Jansohn January 20, 2021

I don't think I'm allowed to use Zoom in my company. Any chance we can do a Microsoft Teams call?

Like
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 20, 2021

Sure.

Send me the MS Teams link and I will join

Like
Robin Jansohn
Robin Jansohn January 20, 2021

EDIT: removed link

Like
Robin Jansohn
Robin Jansohn January 20, 2021

@Thiago Masutti 

I got it working now with the docker DNS resolver. Here is my nginx configuration:

client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony/ {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}
Like
Thiago Masutti
Thiago Masutti
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
January 20, 2021

@Robin Jansohn 

 

I'm almost sure that using location /synchrony/ makes nginx to send all the requests to the default / location, which will then use the Tomcat Synchrony Proxy.

You can confirm by looking at the developer tools and validate requests are sent to /synchrony and not to /synchrony-proxy.

 

I was able to reproduce the issue if I'm using the resolver directive in nginx, so I'm testing things here.

Regards,
Thiago

Like
Reply

Suggest an answer

Log in or Sign up to answer
Confluence
Confluence
DEPLOYMENT TYPE
SERVER
VERSION
7.4.6
TAGS
AUG Leaders

Atlassian Community Events

    See all events