CDN/CloudFront and Atlassian Server apps

[Craig Castle-Mead]

There's a few discussions around speeding up Jira/Confluence etc server installs, so thought it would be beneficial to kick start a discussion and share some learnings, and see what others had tried with what results.

The install the team I'm part of look after is for a global organization with around 15,000 users spread all around the world. We're running Server for both Jira and Confluence and to help get the most out of these and the best performance possible for our users, we looked at how we could use CDN to boost performance.

** disclaimer - below is based on my experience and circumstances only, if you do choose to try anything test in a non-production environment first and no warranty express or implied ** 

Given we're not in control of the application code itself, there was some trial and error on our dev environment, but have been able to get significant benefit from our setup.

Stats since Jan 1st, 2018:

Jira: Of the 689GB traffic initiated by users browsers, 587GB has been served directly from CloudFront, leaving only 102GB needing to go back to the server (origin) ~ 85%

Confluence: Of 185GB total, 160GB has come from CloudFront, 26GB from the server ~ 86%

While the traffic that does make it back to the server is where the server needs to do most work (dynamic content), offloading such a large percentage of the easy traffic has allowed us to get more life out of the infrastructure and licensing we do have. 

NB: Even if you choose to enable no caching whatsoever, enabling a CDN will (should) assist with performance as end users traffic will go from their ISP to the AWS backbone at the closest edge location and then travel along the AWS backbone to your server. 

Given most of the objects that are requested from the server require authentication, you'll need to pass through cookies. There's also heavy use of URI queries/params, so these are passed through as well.

When doing an app or add-on update, likely safest to run a full cache invalidation once complete to ensure no old versions are being served.

Anything not mentioned is using the default setting:

JIRA

1. Path Pattern = /s/*
   
   1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Use Origin Cache Headers
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, cache based on all
2. Path pattern = /secure/projectavatar*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 86400
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. pid
         size
3. Path pattern = /secure/viewavatar*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 86400
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. avatarId
         avatarType
         size
4. Path pattern = /download/resources/*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Default = 86400
      2. Max = 31536000
   4. Forward Cookies = None
   5. Query String Forwarding and Caching = Forward all, cache based on all
5. Path pattern = /robots.txt and /favicon.ico (cause hey, why not)
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 31536000
   4. Forward Cookies = None
   5. Query String Forwarding and Caching = None
6. Default:
   1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
   2. Cache Based on Selected Request Headers = All
   3. Forward Cookies = All
   4. Query String Forwarding and Caching = Forward all, cache based on all

Confluence:

We're just doing the same /s/* for Confluence for now.

If anyone has any feedback, it'd be great to hear how you were able to improve performance. Or, even if you tried something but it didn't work.

CCM

[Craig Castle-Mead]

We have been testing the 8.x line of Jira, and now with 8.5.0 EAP2 out (8.5.x being the Enterprise Release), and some updates to apps to provide support (there was an Indexing change in 8.4.x that seemed to cause some issues), we're back on to testing.

I plan to write up a new Jira 8+ CDN thread shortly to summarize findings.

CCM

[Craig Castle-Mead]

As you may have heard, Atlassian announced CDN support for jira/confluence/Bitbucket (web UI, SSH not suported) at Summit 2019.

The CDN snippet from https://community.atlassian.com/t5/Removed-Posts/Sneak-peek-Improvements-coming-to-Server-and-Data-Center/td-p/1067940 is:

• CDN support: To support better performance for distributed teams, we are introducing content delivery network (CDN) support for our core Data Center offerings. By enabling CDN support, you can accelerate the experience of your remote users as they interact with the UI of our products and reduce peak load on your primary application instances. You can use any CDN you want or a reverse proxy setup. We'll also provide easy integration with AWS CloudFront in updated quick start templates.

I’m very much looking forward to seeing how Atlassian implement the CDN setup and being able to offload our /s/ content again.

https://confluence.atlassian.com/adminjira/preparing-for-jira-8-2-968675818.html has a note about their underlying change to help with the support:

In Jira 7.0 we moved to stateless delivery of JavaScript and CSS resources. Apps that don't use the new APIs for web-resource transforms and conditions may cause static assets to be cached incorrectly, when CDN is enabled. See Stateless web-resource transforms and conditions to make sure your app is using the new APIs for web-resource transforms and conditions. 

We will also provide a health check on the CDN administration screen that will indicate if an app is using deprecated methods. 

These changes will not be included in Jira 8.2, but we're planning to add them in one of the next releases.

Ill be keeping an eye on the release notes for the first CDN version and get to testing ASAP.

CCM

[Craig Castle-Mead]

Hi @M Amine 

I can see from the start of the Origin that you’re pointing CloudFront to a load balancer. If the LB has Sticky sessions on (which you need for data center) then this is why you’re getting no hits (there’s a conflict between the session cookies and the CDN).

you can:

- wait for JIRA 8.3 data center which changes how /s/ content works and use that

- if you’re only running JIRA server and have a single node (which still makes sense with a load balancer to handle ssl offloading and move your ec2 instance off the direct internet), then turn off sticky sessions for the LB as it can only send traffic to one node anyway

CCM

[Craig Castle-Mead]

Hey @Rudiger  - where would the fun (and sleepless nights) be if it was all consistent and 100% clear and unambiguous documentation :)

CCM

[Craig Castle-Mead]

Hi Mohammed,

we have tested the AWS WAF with OWASP top 10 rules from the marketplace, however it quickly started getting false positives as you found. Jira tickets often contain JavaScript code snippet as the description or comment - and WAFs see this as code injection and block it. Still looking for an option that (somehow) knows what’s good code and what’s bad code. 

CCM

[David Barrett _Deloitte_]

Hi @Craig Castle-Mead, many thanks for your post - very handy.

Today I think I found out why some of us were having trouble with CloudFront not caching our content. If you use CloudFront --> ALB --> JIRA, and have stickiness turned on in the target group then objects won't cache. Turning it off meant the AWSALB and Expires cookies were no longer being set, and caching started working.

It would be interesting to see if others find this useful, and if there is any way to have stickiness turned on without it breaking things.

- Dab

[Craig Castle-Mead]

Glad to hear you saw some improvement. 

Re: other items that can be cached, I haven't dug too much further with Confluence yet (the benefits for the work done so far were quite high, so priority wise it dropped down the list), but my method is using the network panel in Chrome (or similar) to look for repeating URL patterns returning the same content, doing some digging online to try and validate my understanding of those objects - what they're used for/when they change/etc, determine what a suitable TTL would be (eg: for project avatars in Jira, if an image is cached a bit too heavily, it might be slightly annoying/confusing to a user but not a show stopper), then test it out in UAT, validate and roll to prod if happy.

/synchrony/ (based on my understanding), is all about the collaborative live editing which is unlikely to suit caching as the data is dynamic. I did however notice that /synchrony-proxy/resources/js/synchrony.min.js is a 166KB JS file without URL parameters etc, seems to be loaded, consistent file size (according to our nginx logs), requested ~ 150 times/hour in our environment, so would be a minimal saving, but every little bit helps.

CCM

[sarath sasi]

Hi All/@Craig_Castle-Mead ,

One major difference I am seeing in the document is not all the traffic is going through the edge server and jira redirects traffic to edge server only  for static assets.

https://confluence.atlassian.com/adminjiraserver/configure-your-cdn-for-jira-data-center-974378841.html

In short A record has not to be changed to the cdn server as per atlassian support.

[Tran Dien Hien]

Hi @Craig Castle-Mead 

Thanks for your sharing. I'm facing a problem related to the fonts (adgs-icons.woff & adgs-icons.ttf). Please see in attached file for more details. Could you please give me some advice to solve that problem?

Thanks 

[Jakub Lazinski]

Hi @Aditya Verma

CDN support is available in JSW 8.3, JSD 4.3 and Confluence 7.0. The documentation is available here: https://confluence.atlassian.com/adminjiraserver/use-a-cdn-with-atlassian-data-center-applications-974378840.html

Thanks,
Jakub

[Aditya Verma]

Hi @Craig Castle-Mead 

Is there any documentation which can be used to start configuring CDN for Jira Server/datacenter versions.

We would certainly like to explore CDN as our Jira and Confluence applications are based in Europe and are used by users all over the globe.

Regards,
Aditya

[M Amine]

Hi @Craig Castle-Mead 

Thank you vm for your quick reply. 

The stickiness is disabled in the Target Group I'm using (please see below) : 

Moreover I have configured all the behaviors you proposed and unfortunately non is used by my CDN as I noticed in AWS stats :

Any advise?

thank you. 

[M Amine]

Hey,

I have configured the "/s/*" behavior (actually "/jira/s/*") and "strangely!" every /s/ request is a miss. 

is there a way to analyse why? 

[Rudiger]

Thanks Craig. I'm a bit lost though (I don't follow the Jira updates as well as I should). Jira 7.0 is not the current release, but they moved to stateless js / css. In theory anyone with 7.x should have been able to get a CDN working if they were self hosted. But I had major issues getting it working. Though I finally did it wasn't worth the tradeoff.

Now we're currently on 8.1, but they are removing stateless on 8.2, but then adding it back in the future? This is confusing as hell.

[Yogesh Khandelwal]

@Craig Castle-Meadany luck with force caching?

[David Yu]

Are you turning on stickyness due to Data Center Edition? What about this dark feature (jira.fixed.cdn.enabled), would that help in routing users to a different ALB that isn't sticky?

[Craig Castle-Mead]

Working on the /s/* issue still with Stickyness on. I've yet to get a helpful reply on how to get cookie based stickyness working in harmony when the application cookies are needed by the CDN. I'm testing a few workaround configs to force caching regardless - will report back.

CCM

[Yogesh Khandelwal]

@Craig Castle-Meadall behaviors with forced cache min/max values are being cached as expected - Does this mean setting the minimum TTL to 0? or can you share your configuration?

[Yogesh Khandelwal]

@RudigerWe have removed the apache now and directly using ALB which is pointing to jira instances.

We are still seeing missed from cloudfront for all the requests.

[Yogesh Khandelwal]

Hi @Craig Castle-Meaddid you get any update from /s/* behavior?

[Craig Castle-Mead]

Hey all,

I spent some time testing this morning and can reproduce this in our environment as well (and, not surprisingly, our cache effectiveness has dropped significantly since we moved to ALB with stickyness on).

With stickyness on (as required) - all behaviors with forced cache min/max values are being cached as expected, however the /s/* behavior which has stickyness on does not, and does not have the set-cookie header at all. With stickyness on, set-cookie exists with an Expires= value as you mentioned, and the object starts being cached.

I've lodged a request with our AWS support team and will see what they come back with and test any suggestions in our UAT enviro.

CCM

[Craig Castle-Mead]

Hey David,

Thats definitely something I’ll check in to on our side and report back. When I originally wrote this we were on JIRA server and ELBs. We’ve since moved to DC and ALBs (with stickysessions), will check in to what’s being cached and report back for all.

CCM

[Craig Castle-Mead]

Hey Zhenis - sorry for the delay, I've been busy/travelling - believe Matt S. has scheduled a call next Tuesday with yourself and Jakub K. to discuss this during our TAM workshop.

CCM

[Yogesh Khandelwal]

we don't have any header set in apache but still we are getting the issue of nothing being cached.

[Rudiger]

Just unset Expires

I'd like to investigate improving the headers but I can't give a timeframe for it.