Confluence Node can't connect to Synchrony node

[Chris Scott-Thomas]

I have followed the guide for moving from Server -> Datacenter edition.

After setting up a synchrony node, my confluence node can not connect to it.

"We couldn't connect to your Synchrony cluster from this Confluence node"

There is nothing in *any* log saying why, which is insanely frustrating.

The only synchrony setting I have in setenv.sh is:

CATALINA_OPTS="-Dsynchrony.service.url=http://FQDN/synchrony/v1"

Though when I curl this I get a 404...

The FQDN is being handled by an nginx reverse proxy.

I can access the synchrony node through a web browser (fqdn/synchrony/heartbeat) and I can curl the same from my workstation, and the confluence node.

curl -v https://FQDN/synchrony/heartbeat
* Trying PROXY_IP...
* Connected to FQDN (PROXY_IP) port 443 (#0)
* found 150 certificates in /etc/ssl/certs/ca-certificates.crt
* found 600 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *FQDN (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: TLS stuff
* start date: Tue, 16 Aug 2016 00:00:00 GMT
* expire date: Wed, 21 Aug 2019 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET /synchrony/heartbeat HTTP/1.1
> Host: FQDN
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.12.2
< Date: Thu, 28 Jun 2018 12:57:04 GMT
< Content-Type: text/plain
< Content-Length: 3
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: content-type, accept, x-token, x-atlassian-mau-ignore
< Access-Control-Allow-Methods: GET, PUT, POST, OPTIONS
< Access-Control-Max-Age: 86400
<
OK
* Connection #0 to host FQDN intact

The only interaction I see from the confluence node to the synchrony node, is a request for a jwt-key:

{"timestamp":"2018-06-28T12:34:43,234Z","level":"INFO","thread":"async-dispatch-15","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"PROXY_IP","uri":"/synchrony/jwt-key","request-method":"get","query-string":null,"x-forwarded-for":"CONF_NODE_IP"},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}

Now I can wget this jwt-key, and its contents are 'dummy public key', which doesn't seem right to me...

Without any decent logging it's really difficult to route out the issue.  Can anyone point me in the right direction here?

[Chris Scott-Thomas]

I will answer this in case anyone else gets stuck.

The official guides states:

Start Confluence on node 1 and pass the following system property to Confluence to tell Confluence where to find your Synchrony cluster.

-Dsynchrony.service.url=http://<confluence-base-url>/synchrony/v1

For example http://yoursite.example.com/synchrony/v1. You must include /v1 on the end of the URL.

This is bad information.  instead you want to have synchrony on an alternate URL, being handled by your load balancer, and point it at that.  Then it works.