It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Synchrony failing - Error Validating JWT

All,  I have been beating my head in to this for the last 2 days and am finally reaching out for help.

This is a greenfield Confluence Data Center install with 2 standalone Synchrony nodes.  The Synchrony nodes are online, and appear to have formed a cluster just fine.

The problem is with Confluence.  When Collaborative editing is enabled, the admin page shows that Synchrony is running, and I can get the heartbeat "OK", but users get the spinning wheel of death when trying to edit a page.

There are several log entries that would seem to be germane.  They are:

(confluence log)

WARN [http-nio-8090-exec-2] [synchrony.service.http.SynchronyResponseHandler] failed Synchrony external changes API call returned 401: {"type":"jwt\/invalid-aud","message":"Error validating JWT"} content-id: 655377 rev: null ancestor: null merges: {"confVersion":"1","trigger":null,"type":"external"} generate-rev: true generate-reset: true

(synchrony log)

{"timestamp":"2018-03-25T23:27:08,244Z","level":"INFO","thread":"async-dispatch-6","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/apps/secret","request-method":"post","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}
{"timestamp":"2018-03-25T23:27:08,254Z","level":"WARN","thread":"async-dispatch-14","logger":"synchrony.http","message":{"synchrony":{"message":"synchrony.http [warn] Secret request made from outside IP whitelist","ip":"10.168.10.110","ns":"synchrony.http","level":"warn"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}
{"timestamp":"2018-03-25T23:27:08,257Z","level":"INFO","thread":"async-dispatch-7","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/apps/secret","request-method":"post","query-string":null},"response":{"status":200},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}

{"timestamp":"2018-03-25T23:27:36,586Z","level":"INFO","thread":"async-dispatch-5","logger":"synchrony.middleware.logging","message":{"synchrony":{"request":{"remote-addr":"10.168.10.110","uri":"/synchrony/v1/data/Synchrony-02ec05b3-8131-3114-941d-d0f736b6bcff/confluence-655377","request-method":"put","query-string":null},"response":{"status":401},"ns":"synchrony.middleware.logging","level":"info","message":"synchrony.middleware.logging [info] null"}},"location":{"class":"synchrony.logging$eval69$fn__73","method":"invoke","line":"0"}}

 

I found the page regarding running the Sec.jar app to discover the JWT keys.  But the KB is extremely poorly written.  It doesn't explain if it's supposed to be ran on a synchrony node or a confluence node (they give different keys), and it seems to be geared for an internal synchrony service vice a cluster set up in an HA environment.

I have tried every variation I can think of to try and get the keys working, but nothing is helping.

Does anyone know how I can fix this?

1 answer

0 votes

Neill,

Can you let me know which KB you're referring to? You're meant to edit the confluence.cfg.xml file in the Confluence Home Directory.

  1. Download the file Sec.jar
  2. Run the following command in a terminal:
    java -jar Sec.jar
    
  3. Stop Confluence
  4. Copy and paste the output into confluence.cfg.xml, we need to replace jwt.private.key and jwt.public.key properties entirely
  5. Start Confluence
  6. Go to Collaborative Editing page, set the mode to Off and wait for Synchrony to stop
  7. Set the mode back to On

This problem is similar to the following bug report, which affected Data Center upgrades:

Let me know if that's what you've tried or if you have any other questions about it.

Regards,

Shannon

Shannon,

Yes, that's the exact procedure I am referring to.

It's not clear which server that is supposed to be ran on.  Running Sec.jar on the confluence server and running it on the Synchrony server give different results.

It doesn't really matter though, neither works after updating the confluence.cfg.xml on the confluence server.

Hi Neill,

The keys are found on each node within their respective confluence.cfg.xml files. 

  • For versions earlier than Confluence 6.5, the Synchrony startup command will need to be updated as well.
  • For Confluence 6.5 and higher, they're just updated in Confluence, but all nodes need to have the same key.

However, since you're getting the invalid-aud message, that means that the Synchrony URL is incorrect on either the Confluence side or the Synchrony side.

  • In Confluence, it should be set to CONFLUENCE-URL/synchrony/v1
  • In Synchrony, it should be set to CONFLUENCE-URL/synchrony

You can have a look at Configuring Synchrony for Data Center for further information on that.

Regards,

Shannon

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published Thursday in Confluence

Confluence CVEs and common questions

Two vulnerabilities have been published for Confluence Server and Data Center recently: March 20, 2019 CVE-2019-3395 / CVE-2019-3396 April 17, 2019 CVE-2019-3398 The goal of this article is...

257 views 0 12
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you