Synchrony failing - Error Validating JWT

[Neill Thornton]

All,  I have been beating my head in to this for the last 2 days and am finally reaching out for help.

This is a greenfield Confluence Data Center install with 2 standalone Synchrony nodes.  The Synchrony nodes are online, and appear to have formed a cluster just fine.

The problem is with Confluence.  When Collaborative editing is enabled, the admin page shows that Synchrony is running, and I can get the heartbeat "OK", but users get the spinning wheel of death when trying to edit a page.

There are several log entries that would seem to be germane.  They are:

(confluence log)

WARN [http-nio-8090-exec-2] [synchrony.service.http.SynchronyResponseHandler] failed Synchrony external changes API call returned 401: {"type":"jwt\/invalid-aud","message":"Error validating JWT"} content-id: 655377 rev: null ancestor: null merges: {"confVersion":"1","trigger":null,"type":"external"} generate-rev: true generate-reset: true

(synchrony log)

{"timestamp":"2018-03-25T23:27:08,244Z","level":"INFO","thread":"async-dispatch-6","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/apps/secret","request-method":"post","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}
{"timestamp":"2018-03-25T23:27:08,254Z","level":"WARN","thread":"async-dispatch-14","logger":"synchrony.http","message":{"synchrony":{"message":"synchrony.http [warn] Secret request made from outside IP whitelist","ip":"10.168.10.110","ns":"synchrony.http","level":"warn"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}
{"timestamp":"2018-03-25T23:27:08,257Z","level":"INFO","thread":"async-dispatch-7","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/apps/secret","request-method":"post","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}

{"timestamp":"2018-03-25T23:27:36,586Z","level":"INFO","thread":"async-dispatch-5","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/data/Synchrony-02ec05b3-8131-3114-941d-d0f736b6bcff/confluence-655377","request-method":"put","query-string":null},"response":{"status":401},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}

I found the page regarding running the Sec.jar app to discover the JWT keys.  But the KB is extremely poorly written.  It doesn't explain if it's supposed to be ran on a synchrony node or a confluence node (they give different keys), and it seems to be geared for an internal synchrony service vice a cluster set up in an HA environment.

I have tried every variation I can think of to try and get the keys working, but nothing is helping.

Does anyone know how I can fix this?

[Shannon S]

Neill,

Can you let me know which KB you're referring to? You're meant to edit the confluence.cfg.xml file in the Confluence Home Directory.

1. Download the file Sec.jar
2. Run the following command in a terminal:

   java -jar Sec.jar
   

3. Stop Confluence
4. Copy and paste the output into confluence.cfg.xml, we need to replace jwt.private.key and jwt.public.key properties entirely
5. Start Confluence
6. Go to Collaborative Editing page, set the mode to Off and wait for Synchrony to stop
7. Set the mode back to On

This problem is similar to the following bug report, which affected Data Center upgrades:

• https://jira.atlassian.com/browse/CONFSERVER-45657

Let me know if that's what you've tried or if you have any other questions about it.

Regards,

Shannon

[Neill Thornton]

Shannon,

Yes, that's the exact procedure I am referring to.

It's not clear which server that is supposed to be ran on.  Running Sec.jar on the confluence server and running it on the Synchrony server give different results.

It doesn't really matter though, neither works after updating the confluence.cfg.xml on the confluence server.

[Shannon S]

Hi Neill,

The keys are found on each node within their respective confluence.cfg.xml files. 

• For versions earlier than Confluence 6.5, the Synchrony startup command will need to be updated as well.
• For Confluence 6.5 and higher, they're just updated in Confluence, but all nodes need to have the same key.

However, since you're getting the invalid-aud message, that means that the Synchrony URL is incorrect on either the Confluence side or the Synchrony side.

• In Confluence, it should be set to CONFLUENCE-URL/synchrony/v1
• In Synchrony, it should be set to CONFLUENCE-URL/synchrony

You can have a look at Configuring Synchrony for Data Center for further information on that.

Regards,

Shannon