CDN/CloudFront and Atlassian Server apps

[Craig Castle-Mead]

There's a few discussions around speeding up Jira/Confluence etc server installs, so thought it would be beneficial to kick start a discussion and share some learnings, and see what others had tried with what results.

The install the team I'm part of look after is for a global organization with around 15,000 users spread all around the world. We're running Server for both Jira and Confluence and to help get the most out of these and the best performance possible for our users, we looked at how we could use CDN to boost performance.

** disclaimer - below is based on my experience and circumstances only, if you do choose to try anything test in a non-production environment first and no warranty express or implied ** 

Given we're not in control of the application code itself, there was some trial and error on our dev environment, but have been able to get significant benefit from our setup.

Stats since Jan 1st, 2018:

Jira: Of the 689GB traffic initiated by users browsers, 587GB has been served directly from CloudFront, leaving only 102GB needing to go back to the server (origin) ~ 85%

Confluence: Of 185GB total, 160GB has come from CloudFront, 26GB from the server ~ 86%

While the traffic that does make it back to the server is where the server needs to do most work (dynamic content), offloading such a large percentage of the easy traffic has allowed us to get more life out of the infrastructure and licensing we do have. 

NB: Even if you choose to enable no caching whatsoever, enabling a CDN will (should) assist with performance as end users traffic will go from their ISP to the AWS backbone at the closest edge location and then travel along the AWS backbone to your server. 

Given most of the objects that are requested from the server require authentication, you'll need to pass through cookies. There's also heavy use of URI queries/params, so these are passed through as well.

When doing an app or add-on update, likely safest to run a full cache invalidation once complete to ensure no old versions are being served.

Anything not mentioned is using the default setting:

JIRA

1. Path Pattern = /s/*
   
   1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Use Origin Cache Headers
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, cache based on all
2. Path pattern = /secure/projectavatar*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 86400
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. pid
         size
3. Path pattern = /secure/viewavatar*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 86400
   4. Forward Cookies = All
   5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. avatarId
         avatarType
         size
4. Path pattern = /download/resources/*
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Default = 86400
      2. Max = 31536000
   4. Forward Cookies = None
   5. Query String Forwarding and Caching = Forward all, cache based on all
5. Path pattern = /robots.txt and /favicon.ico (cause hey, why not)
   1. Allowed HTTP methods = GET, HEAD
   2. Cache Based on Selected Request Headers = Whitelist
   3. Object Caching = Customize
      1. Min/Max/Default = 31536000
   4. Forward Cookies = None
   5. Query String Forwarding and Caching = None
6. Default:
   1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
   2. Cache Based on Selected Request Headers = All
   3. Forward Cookies = All
   4. Query String Forwarding and Caching = Forward all, cache based on all

Confluence:

We're just doing the same /s/* for Confluence for now.

If anyone has any feedback, it'd be great to hear how you were able to improve performance. Or, even if you tried something but it didn't work.

CCM

[Itesh Shah]

Hi, I am trying to do the same to get better access speed for Jira. What dud you use for CDN and how are you hosting your instances

[Craig Castle-Mead]

Hi @Itesh Shah

We're using AWS CloudFront as the CDN.

What do you mean by your "how are you hosting your instances?" They're a combination of M4 and M5 EC2 instances in AWS eu-central-1 - this made most sense from a compliance and central geographical point of view

CCM

[Itesh Shah]

Hi,

Thank you for the revert to my query.

How are you hosting your instances I meant wether they are "On-prem in a datacenter" or "Hosting in the cloud (AWS/Google/etc) sorry that I did not clarify what i meant. Thank you though for letting me know that you are doing it in AWS.

We are having an "On-prem" situation and trying to use cloud front.
Any special changes that needs to be done on Jira other than enabling the dark feature? and any special attention that I need to pay for CloudFront configs?

Regards,

Itesh

[Craig Castle-Mead]

Hi Itesh,

Sorry for the belated reply - totally missed the notification.

We did not change anything in the Jira config at all - it was just a bunch of network inspection in browsers and looking for URL patterns that returned consistent content and mapping these to behaviors with varying levels of caching. The default behavior enforces no cache so just sends the entire request (all headers, cookies, query strings, methods) back to the origin as if the CDN didn't exist.

CCM

[Robert Beckman]

This is a very informative list, but I'm a little stuck as to what headers you are whitelisting in the caching settings.

[Robert Beckman]

To answer my own question, whitelisting "Origin" headers seems to do most of the trick, but I'm having issues with woff and ttf files for some reason. They end up cancelling entirely.

[Craig Castle-Mead]

Hey Robert,

Not sure what could be happening with your ttf/woff's - maybe worth putting a rule just for *.woff and *.ttf at the top of your rules and forcing them through without any cache to check if they at least load, and if that works, you can change the behavior so they then should be cached - at least this way you know exactly which rule you're diagnosing (actually, it may be worth adding a custom header to each of your behaviors (rule = 1, rule = 2 etc etc) so that you can view the headers of each file and it will tell you which rule the file is using)

CCM

[Robert Beckman]

Digging in a little more on the .woff and .ttf, they're transferring correctly in some browsers, but CloudFront is returning a 502 in others, even if I set them up in a passthrough rule.

For the browsers where they do transfer, they fail to load/display due to CORS, seeing as how their Origin is my Jira instance but they're being downloaded through cloudfront.net. I'm currently working that angle, but have been unable to set Access-Control-Allow-Origin headers on the requests.

[M Amine]

Thank you vm @Craig Castle-Mead for the post. 

Tried to do the samething but I'm wondering how would I disable any traffic not getting from the cloudfront distribution? 

In my case I'm using an EC2 instance with a reverse-proxy sending traffic to Jira & Confluence. 

thanks

MKA

[Craig Castle-Mead]

Hey Mohammed,

To ensure only CloudFront traffic is getting through to your environment, you'll want to have the security group automatically update based on the IP ranges CloudFront uses. As these can (and do) change, AWS have an SNS topic you can subscribe, this can then trigger a Lambda function that updates the security group inbound rules (write up at https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/ )

PS. Even if you only have a single EC2 instance, I'd suggest putting a load balancer in front it and only allowing 80/443 to the load balancer from CloudFront and then only allowing 80/443 to the EC2 instance from the load balancer.

CCM

[M Amine]

Thank you for your reply @Craig Castle-Mead. 

Let me give you come more details about the infrastructure I'm managing:

• Cloudfront
• Origin: EC2 Instance in a public Subnet
  • NACL & SG (allowing only 80/443) + will add the lambda function (thanks ;) )
  • Nginx (reverse Proxy): forwarding 80(http)->443(https) + SSL + ... etc
  • Crowd
  • Jira
  • Confluence
  • ...
• RDS on a private Subnet

Do you think that I need to add the LB?

Besides, I'm still not able to deploy the CDN. I went through all the steps here but still getting 502 error page. I'm missing something that I cann't figure out !

Here is my configuration:

CNAME: I'm using 'xxx.domainname.com' for which I have created a CNAME record in my host (OVH).

I have a generated a SSL certificate that I have uploaded to ACM and used in cloudfront. 

While troubleshooting I have done the following:

• Used an SSL checker (https://www.sslchecker.com/sslchecker) which tells me that the certificate and the chain were find but not the root. BTW other online SSL checkers say that everything is ok. Don't know if the issue is coming from here. Amazon CM finds that the private, certificate and the chain are OK. 
• Used Openssl
  • openssl s_client –connect domainname:443 –servername domainname ==> is OK
  • openssl s_client –connect domainname:443 ==> is giving me SSL HANDSHAKE ERROR. But Cloudfront is not configured to allow non SNI clients. So I think it is ok
• Checked my domaine with https://www.ssllabs.com/ssltest/ for ciphers & protocols. Everything seem to be ok. I can give more details if necessary

Really cann't find what I'm missing !!

Thank you for your help

kind regards

[M Amine]

Just to follow up and share some information that i've just got from Amazon support: the ssl check I did was apparently the wrong one. I had to use the EC2 custom domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com)and not my custom domain name (xxx.mydomainname.com).

So I'm now wondering if I have to generate a certificate for the ec2 domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com) and use that for both cloudfront and my Nginx Server (installed in my EC2 instance)? and just delete (and forget about) my custom domain name SSL certificate?

And what shall I use as base URL for Jira and confluence?

@Craig Castle-Mead for the "Path Pattern = /s/*", you said that you used "Cache Based on Selected Request Headers = Whitelist". Can you please share the whitelist? and also share what you are using as a base URL for Jira (is it your domain name of the ec2 custom domain name)

thank you vm

[M Amine]

Just for followup, it is impossible to generate SSL certificate using letsencrypt for ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com !!

[Craig Castle-Mead]

Hey Mohammed,

1. /s/* behaviour screenshot below
2. If we assume your end users are going to access jira from https://jira.example.org
   
   1. Get a wildcard SSL for example.org
   2. Deploy using AWS Certificate Manager
   3. CloudFront
      1. Use the wildcard cert from Cert Manager
      2. CNAME should be jira.example.org
      3. Origin should be the CNAME of the ELB (eg: jira-example-org-123456.us-west-1.elb.amazonaws.com)
   4. ELB
      1. Use the wildcard cert from Cert Manager
      2. Point to HTTPS listener to HTTP port on the EC2 instances (SSL offloading)
      3. We also CNAME jira-elb.example.org to the ELB. We've had SSL issues when using BobSwift CLI going through CloudFront but when pointing BobSwift to the -elb hostname all works OK
   5. Nginx
      1. No SSL needed
   6. Jira app
      1. No SSL needed
      2. BaseURL is https://jira.example.org

CCM

[M Amine]

Thank you vm for sharing all these info. 

I'm wondering what are you using as a 'proxyName' in the 'server.xml'. Are you using jira.example.org? or jira-elb.example.org? 

kind regards

[Craig Castle-Mead]

Server.xml proxyname is JIRA.example.org

[M Amine]

Thank you vm @Craig Castle-Mead. It's finally working. The key takeaway point is using the ALB. It really changed everything. 

I had to rebuild a big portion of my infrastructure as I was mainly relying on Nginx to do what I'm now doing with the ALB. 

Really thank you vm for everything. 

Are you using a WAF with your distribution? 

I have just used a template given by Amazon and found that some Jira gadgets (like "Assigned to me" gadget) is not able to display the content

'An unknown error occurred while trying to perform a search'.

After investigation I found that the SQL Injection rule was blocking the content because the JQL query is transmitted in the URI. 

best regards

[Craig Castle-Mead]

Hi Mohammed,

we have tested the AWS WAF with OWASP top 10 rules from the marketplace, however it quickly started getting false positives as you found. Jira tickets often contain JavaScript code snippet as the description or comment - and WAFs see this as code injection and block it. Still looking for an option that (somehow) knows what’s good code and what’s bad code. 

CCM

[lbailey79]

Thank you @Craig Castle-Mead, this worked for me and our Confluence users are definitely benefiting from the improved performance.

We had issues initially which I think was due to using a 3rd party SSO provider, and I needed to recreate the CloudFront distribution from scratch before it would work, but all good now.

I have posed this to Atlassian support as well - Do you think there are other Confluence directories which would benefit from being served from CloudFront? E.g. is the /synchrony/* directory serving static JS content?

Thanks, Liam

[Craig Castle-Mead]

Glad to hear you saw some improvement. 

Re: other items that can be cached, I haven't dug too much further with Confluence yet (the benefits for the work done so far were quite high, so priority wise it dropped down the list), but my method is using the network panel in Chrome (or similar) to look for repeating URL patterns returning the same content, doing some digging online to try and validate my understanding of those objects - what they're used for/when they change/etc, determine what a suitable TTL would be (eg: for project avatars in Jira, if an image is cached a bit too heavily, it might be slightly annoying/confusing to a user but not a show stopper), then test it out in UAT, validate and roll to prod if happy.

/synchrony/ (based on my understanding), is all about the collaborative live editing which is unlikely to suit caching as the data is dynamic. I did however notice that /synchrony-proxy/resources/js/synchrony.min.js is a 166KB JS file without URL parameters etc, seems to be loaded, consistent file size (according to our nginx logs), requested ~ 150 times/hour in our environment, so would be a minimal saving, but every little bit helps.

CCM

[Rudiger]

Thanks for your guide. I successfully implemented it and Jira is working fine but nothing is being cached. Every single resource is 'Miss from cloudfront', is anything under /s/* ever cached?

[Craig Castle-Mead]

Hey Neil,

The way I start testing is finding a call for a /s/ item, opening it in a new tab and refreshing a few times and monitoring the Header x-cache and see if it changes from Miss to Hit. I also added the x-cache header as a custom column in the network panel, makes it easy to see at a glance how many items are getting hits.

If a single request from /s/ never changes from Miss to Hit, revalidate the CloudFront Behaviors.

CCM

[Rudiger]

Hi Craig, thanks for getting back to me. Looking at the assets under /s/* they all have "Expires" as the time it was requested. I can't see any documentation around changing this, is this something you changed or override on yours either at apache / tomcat conf or CDN?

I've also removed this at the apache level and it's still not cached. I also changed the object caching to custom and set it to 86400, still no caching. I believe it's because each load it sets a new cookie, even for just the JS files.

[Craig Castle-Mead]

Hey Neil,

Without digging deeper, I'm out of suggestions at the moment. Given there's no private messaging on here, do you want to find me on LinkedIn (I have a unique name so should be easy to find). Drop me a message and happy to see if we can line up a quick chat to go through it.

CCM

[Yogesh Khandelwal]

I am also facing the same issue. Can we sync up on linkedin?

[Yogesh Khandelwal]

Hi @Craig Castle-Mead I have enabled the same settings as yours but i am not seeing any hits in monitoring to cloudfronts. I have cloudfront then ALB then nginx and app server running. All are on 443.

Can you help me to find out what could be the cause?

[Rudiger]

So I believe I've found the issue. Jira sets an Expires header to the current time, hence it is always expired and will never get a hit unless done on the same second.

Unsetting it in the Apache or Nginx config will then allow it to be cached however this makes most requests served from local memory, only when someone else hits the Jira server is it served by CloudFront (and from then on local). This does make the page very fast to load comparatively. I want to spend more time on this to make it better however it just seems very hacky.

I don't like the idea of this personally because if there is an update, invalidating the CloudFront cache won't achieve anything.

Personally I think this is very bad engineering. I can understand having the need for some assets to be requested each time but from what I can tell, in Jira at least, a blanket rule across all assets, static or otherwise, to set to expire immediately.

[Yogesh Khandelwal]

Thanks Neil. Can you please let me know what settings i need to unset in apace or nginx? i have ALB as well on top of apache.

[Rudiger]

Just unset Expires

I'd like to investigate improving the headers but I can't give a timeframe for it.

[Yogesh Khandelwal]

we don't have any header set in apache but still we are getting the issue of nothing being cached.

[Yogesh Khandelwal]

@RudigerWe have removed the apache now and directly using ALB which is pointing to jira instances.

We are still seeing missed from cloudfront for all the requests.

[Zhenis Beisekov]

Hi @Craig Castle-Mead
Thank you very much for sharing your experience. 

Could you please provide a little bit more input on:
- what locations are your users located? where is the server?
- what is the latency between different locations and the server?
- what apps/plugins do you have installed?
- how many users are concurrently online?

I ask, because I would like to figure out how more of Atlassian users can benefit.

[Craig Castle-Mead]

Hey Zhenis - sorry for the delay, I've been busy/travelling - believe Matt S. has scheduled a call next Tuesday with yourself and Jakub K. to discuss this during our TAM workshop.

CCM

[David Barrett (Deloitte)]

Hi @Craig Castle-Mead, many thanks for your post - very handy.

Today I think I found out why some of us were having trouble with CloudFront not caching our content. If you use CloudFront --> ALB --> JIRA, and have stickiness turned on in the target group then objects won't cache. Turning it off meant the AWSALB and Expires cookies were no longer being set, and caching started working.

It would be interesting to see if others find this useful, and if there is any way to have stickiness turned on without it breaking things.

- Dab

[Craig Castle-Mead]

Hey David,

Thats definitely something I’ll check in to on our side and report back. When I originally wrote this we were on JIRA server and ELBs. We’ve since moved to DC and ALBs (with stickysessions), will check in to what’s being cached and report back for all.

CCM

[Craig Castle-Mead]

Hey all,

I spent some time testing this morning and can reproduce this in our environment as well (and, not surprisingly, our cache effectiveness has dropped significantly since we moved to ALB with stickyness on).

With stickyness on (as required) - all behaviors with forced cache min/max values are being cached as expected, however the /s/* behavior which has stickyness on does not, and does not have the set-cookie header at all. With stickyness on, set-cookie exists with an Expires= value as you mentioned, and the object starts being cached.

I've lodged a request with our AWS support team and will see what they come back with and test any suggestions in our UAT enviro.

CCM

[Yogesh Khandelwal]

Hi @Craig Castle-Meaddid you get any update from /s/* behavior?

[Yogesh Khandelwal]

@Craig Castle-Meadall behaviors with forced cache min/max values are being cached as expected - Does this mean setting the minimum TTL to 0? or can you share your configuration?

[Craig Castle-Mead]

Working on the /s/* issue still with Stickyness on. I've yet to get a helpful reply on how to get cookie based stickyness working in harmony when the application cookies are needed by the CDN. I'm testing a few workaround configs to force caching regardless - will report back.

CCM

[David Yu]

Are you turning on stickyness due to Data Center Edition? What about this dark feature (jira.fixed.cdn.enabled), would that help in routing users to a different ALB that isn't sticky?

[Yogesh Khandelwal]

@Craig Castle-Meadany luck with force caching?

[Craig Castle-Mead]

As you may have heard, Atlassian announced CDN support for jira/confluence/Bitbucket (web UI, SSH not suported) at Summit 2019.

The CDN snippet from https://community.atlassian.com/t5/Removed-Posts/Sneak-peek-Improvements-coming-to-Server-and-Data-Center/td-p/1067940 is:

• CDN support: To support better performance for distributed teams, we are introducing content delivery network (CDN) support for our core Data Center offerings. By enabling CDN support, you can accelerate the experience of your remote users as they interact with the UI of our products and reduce peak load on your primary application instances. You can use any CDN you want or a reverse proxy setup. We'll also provide easy integration with AWS CloudFront in updated quick start templates.

I’m very much looking forward to seeing how Atlassian implement the CDN setup and being able to offload our /s/ content again.

https://confluence.atlassian.com/adminjira/preparing-for-jira-8-2-968675818.html has a note about their underlying change to help with the support:

In Jira 7.0 we moved to stateless delivery of JavaScript and CSS resources. Apps that don't use the new APIs for web-resource transforms and conditions may cause static assets to be cached incorrectly, when CDN is enabled. See Stateless web-resource transforms and conditions to make sure your app is using the new APIs for web-resource transforms and conditions. 

We will also provide a health check on the CDN administration screen that will indicate if an app is using deprecated methods. 

These changes will not be included in Jira 8.2, but we're planning to add them in one of the next releases.

Ill be keeping an eye on the release notes for the first CDN version and get to testing ASAP.

CCM

[Rudiger]

Thanks Craig. I'm a bit lost though (I don't follow the Jira updates as well as I should). Jira 7.0 is not the current release, but they moved to stateless js / css. In theory anyone with 7.x should have been able to get a CDN working if they were self hosted. But I had major issues getting it working. Though I finally did it wasn't worth the tradeoff.

Now we're currently on 8.1, but they are removing stateless on 8.2, but then adding it back in the future? This is confusing as hell.

[Craig Castle-Mead]

Hey @Rudiger  - where would the fun (and sleepless nights) be if it was all consistent and 100% clear and unambiguous documentation :)

CCM

[M Amine]

Hey,

I have configured the "/s/*" behavior (actually "/jira/s/*") and "strangely!" every /s/ request is a miss. 

is there a way to analyse why? 

[Craig Castle-Mead]

Hi @M Amine 

I can see from the start of the Origin that you’re pointing CloudFront to a load balancer. If the LB has Sticky sessions on (which you need for data center) then this is why you’re getting no hits (there’s a conflict between the session cookies and the CDN).

you can:

- wait for JIRA 8.3 data center which changes how /s/ content works and use that

- if you’re only running JIRA server and have a single node (which still makes sense with a load balancer to handle ssl offloading and move your ec2 instance off the direct internet), then turn off sticky sessions for the LB as it can only send traffic to one node anyway

CCM

[M Amine]

Hi @Craig Castle-Mead 

Thank you vm for your quick reply. 

The stickiness is disabled in the Target Group I'm using (please see below) : 

Moreover I have configured all the behaviors you proposed and unfortunately non is used by my CDN as I noticed in AWS stats :

Any advise?

thank you. 

[Aditya Verma]

Hi @Craig Castle-Mead 

Is there any documentation which can be used to start configuring CDN for Jira Server/datacenter versions.

We would certainly like to explore CDN as our Jira and Confluence applications are based in Europe and are used by users all over the globe.

Regards,
Aditya

[Jakub Lazinski]

Hi @Aditya Verma

CDN support is available in JSW 8.3, JSD 4.3 and Confluence 7.0. The documentation is available here: https://confluence.atlassian.com/adminjiraserver/use-a-cdn-with-atlassian-data-center-applications-974378840.html

Thanks,
Jakub

[Craig Castle-Mead]

We have been testing the 8.x line of Jira, and now with 8.5.0 EAP2 out (8.5.x being the Enterprise Release), and some updates to apps to provide support (there was an Indexing change in 8.4.x that seemed to cause some issues), we're back on to testing.

I plan to write up a new Jira 8+ CDN thread shortly to summarize findings.

CCM

[Tran Dien Hien]

Hi @Craig Castle-Mead 

Thanks for your sharing. I'm facing a problem related to the fonts (adgs-icons.woff & adgs-icons.ttf). Please see in attached file for more details. Could you please give me some advice to solve that problem?

Thanks 

[sarath sasi]

Hi All/@Craig_Castle-Mead ,

One major difference I am seeing in the document is not all the traffic is going through the edge server and jira redirects traffic to edge server only  for static assets.

https://confluence.atlassian.com/adminjiraserver/configure-your-cdn-for-jira-data-center-974378841.html

In short A record has not to be changed to the cdn server as per atlassian support.