It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

CDN/CloudFront and Atlassian Server apps

Craig Castle-Mead Community Leader Feb 18, 2018

There's a few discussions around speeding up Jira/Confluence etc server installs, so thought it would be beneficial to kick start a discussion and share some learnings, and see what others had tried with what results.

The install the team I'm part of look after is for a global organization with around 15,000 users spread all around the world. We're running Server for both Jira and Confluence and to help get the most out of these and the best performance possible for our users, we looked at how we could use CDN to boost performance.

 

** disclaimer - below is based on my experience and circumstances only, if you do choose to try anything test in a non-production environment first and no warranty express or implied ** 

 

Given we're not in control of the application code itself, there was some trial and error on our dev environment, but have been able to get significant benefit from our setup.

Stats since Jan 1st, 2018:

Jira: Of the 689GB traffic initiated by users browsers, 587GB has been served directly from CloudFront, leaving only 102GB needing to go back to the server (origin) ~ 85%

Confluence: Of 185GB total, 160GB has come from CloudFront, 26GB from the server ~ 86%

 

While the traffic that does make it back to the server is where the server needs to do most work (dynamic content), offloading such a large percentage of the easy traffic has allowed us to get more life out of the infrastructure and licensing we do have. 

 

NB: Even if you choose to enable no caching whatsoever, enabling a CDN will (should) assist with performance as end users traffic will go from their ISP to the AWS backbone at the closest edge location and then travel along the AWS backbone to your server. 

 

Given most of the objects that are requested from the server require authentication, you'll need to pass through cookies. There's also heavy use of URI queries/params, so these are passed through as well.

When doing an app or add-on update, likely safest to run a full cache invalidation once complete to ensure no old versions are being served.

Anything not mentioned is using the default setting:

JIRA

  1. Path Pattern = /s/*
    1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Use Origin Cache Headers
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, cache based on all
  2. Path pattern = /secure/projectavatar*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 86400
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. pid
        size
  3. Path pattern = /secure/viewavatar*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 86400
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. avatarId
        avatarType
        size
  4. Path pattern = /download/resources/*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Default = 86400
      2. Max = 31536000
    4. Forward Cookies = None
    5. Query String Forwarding and Caching = Forward all, cache based on all
  5. Path pattern = /robots.txt and /favicon.ico (cause hey, why not)
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 31536000
    4. Forward Cookies = None
    5. Query String Forwarding and Caching = None
  6. Default:
    1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
    2. Cache Based on Selected Request Headers = All
    3. Forward Cookies = All
    4. Query String Forwarding and Caching = Forward all, cache based on all

 

Confluence:

We're just doing the same /s/* for Confluence for now.

 

If anyone has any feedback, it'd be great to hear how you were able to improve performance. Or, even if you tried something but it didn't work.

 

CCM

 

9 comments

Itesh Shah Apr 30, 2018

Hi, I am trying to do the same to get better access speed for Jira. What dud you use for CDN and how are you hosting your instances

Craig Castle-Mead Community Leader May 02, 2018

Hi @Itesh Shah

We're using AWS CloudFront as the CDN.

What do you mean by your "how are you hosting your instances?" They're a combination of M4 and M5 EC2 instances in AWS eu-central-1 - this made most sense from a compliance and central geographical point of view

 

CCM

Itesh Shah May 02, 2018

Hi,

Thank you for the revert to my query.

How are you hosting your instances I meant wether they are "On-prem in a datacenter" or "Hosting in the cloud (AWS/Google/etc) sorry that I did not clarify what i meant. Thank you though for letting me know that you are doing it in AWS.


We are having an "On-prem" situation and trying to use cloud front.
Any special changes that needs to be done on Jira other than enabling the dark feature? and any special attention that I need to pay for CloudFront configs?

 

Regards,

Itesh

Craig Castle-Mead Community Leader Jun 20, 2018

Hi Itesh,

Sorry for the belated reply - totally missed the notification.

We did not change anything in the Jira config at all - it was just a bunch of network inspection in browsers and looking for URL patterns that returned consistent content and mapping these to behaviors with varying levels of caching. The default behavior enforces no cache so just sends the entire request (all headers, cookies, query strings, methods) back to the origin as if the CDN didn't exist.

CCM

Robert Beckman Jun 13, 2018

This is a very informative list, but I'm a little stuck as to what headers you are whitelisting in the caching settings.

Robert Beckman Jun 20, 2018

To answer my own question, whitelisting "Origin" headers seems to do most of the trick, but I'm having issues with woff and ttf files for some reason. They end up cancelling entirely.

Craig Castle-Mead Community Leader Jul 04, 2018

Hey Robert,

Not sure what could be happening with your ttf/woff's - maybe worth putting a rule just for *.woff and *.ttf at the top of your rules and forcing them through without any cache to check if they at least load, and if that works, you can change the behavior so they then should be cached - at least this way you know exactly which rule you're diagnosing (actually, it may be worth adding a custom header to each of your behaviors (rule = 1, rule = 2 etc etc) so that you can view the headers of each file and it will tell you which rule the file is using)

CCM

Robert Beckman Jul 10, 2018 • edited

Digging in a little more on the .woff and .ttf, they're transferring correctly in some browsers, but CloudFront is returning a 502 in others, even if I set them up in a passthrough rule.

For the browsers where they do transfer, they fail to load/display due to CORS, seeing as how their Origin is my Jira instance but they're being downloaded through cloudfront.net. I'm currently working that angle, but have been unable to set Access-Control-Allow-Origin headers on the requests.

Mohammed Amine Community Leader Jul 04, 2018

Thank you vm @Craig Castle-Mead for the post. 

Tried to do the samething but I'm wondering how would I disable any traffic not getting from the cloudfront distribution? 

In my case I'm using an EC2 instance with a reverse-proxy sending traffic to Jira & Confluence. 

thanks

MKA

Craig Castle-Mead Community Leader Jul 04, 2018

Hey Mohammed,

To ensure only CloudFront traffic is getting through to your environment, you'll want to have the security group automatically update based on the IP ranges CloudFront uses. As these can (and do) change, AWS have an SNS topic you can subscribe, this can then trigger a Lambda function that updates the security group inbound rules (write up at https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/ )

PS. Even if you only have a single EC2 instance, I'd suggest putting a load balancer in front it and only allowing 80/443 to the load balancer from CloudFront and then only allowing 80/443 to the EC2 instance from the load balancer.

 

CCM

Mohammed Amine Community Leader Jul 04, 2018

Thank you for your reply @Craig Castle-Mead

Let me give you come more details about the infrastructure I'm managing:

  • Cloudfront
  • Origin: EC2 Instance in a public Subnet
    • NACL & SG (allowing only 80/443) + will add the lambda function (thanks ;) )
    • Nginx (reverse Proxy): forwarding 80(http)->443(https) + SSL + ... etc
    • Crowd
    • Jira
    • Confluence
    • ...
  • RDS on a private Subnet

Do you think that I need to add the LB?

Besides, I'm still not able to deploy the CDN. I went through all the steps here but still getting 502 error page. I'm missing something that I cann't figure out !


Here is my configuration:

 

Capture d’écran 2018-07-04 à 17.17.50.pngCNAME: I'm using 'xxx.domainname.com' for which I have created a CNAME record in my host (OVH).

Capture d’écran 2018-07-04 à 17.18.01.pngCapture d’écran 2018-07-04 à 17.18.52.png

I have a generated a SSL certificate that I have uploaded to ACM and used in cloudfront. 

While troubleshooting I have done the following:

  • Used an SSL checker (https://www.sslchecker.com/sslchecker) which tells me that the certificate and the chain were find but not the root. BTW other online SSL checkers say that everything is ok. Don't know if the issue is coming from here. Amazon CM finds that the private, certificate and the chain are OK. 
  • Used Openssl
    • openssl s_client –connect domainname:443 –servername domainname ==> is OK
    • openssl s_client –connect domainname:443 ==> is giving me SSL HANDSHAKE ERROR. But Cloudfront is not configured to allow non SNI clients. So I think it is ok
  • Checked my domaine with https://www.ssllabs.com/ssltest/ for ciphers & protocols. Everything seem to be ok. I can give more details if necessary

Really cann't find what I'm missing !!

Thank you for your help

kind regards

Mohammed Amine Community Leader Jul 05, 2018 • edited

Just to follow up and share some information that i've just got from Amazon support: the ssl check I did was apparently the wrong one. I had to use the EC2 custom domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com)and not my custom domain name (xxx.mydomainname.com).

So I'm now wondering if I have to generate a certificate for the ec2 domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com) and use that for both cloudfront and my Nginx Server (installed in my EC2 instance)? and just delete (and forget about) my custom domain name SSL certificate?

And what shall I use as base URL for Jira and confluence?

@Craig Castle-Mead for the "Path Pattern = /s/*", you said that you used "Cache Based on Selected Request Headers = Whitelist". Can you please share the whitelist? and also share what you are using as a base URL for Jira (is it your domain name of the ec2 custom domain name)

thank you vm

Mohammed Amine Community Leader Jul 05, 2018

Just for followup, it is impossible to generate SSL certificate using letsencrypt for ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com !!

Craig Castle-Mead Community Leader Jul 05, 2018

Hey Mohammed,

  1. /s/* behaviour screenshot below
  2. If we assume your end users are going to access jira from https://jira.example.org
    1. Get a wildcard SSL for example.org
    2. Deploy using AWS Certificate Manager
    3. CloudFront
      1. Use the wildcard cert from Cert Manager
      2. CNAME should be jira.example.org
      3. Origin should be the CNAME of the ELB (eg: jira-example-org-123456.us-west-1.elb.amazonaws.com)
    4. ELB
      1. Use the wildcard cert from Cert Manager
      2. Point to HTTPS listener to HTTP port on the EC2 instances (SSL offloading)
      3. We also CNAME jira-elb.example.org to the ELB. We've had SSL issues when using BobSwift CLI going through CloudFront but when pointing BobSwift to the -elb hostname all works OK
    5. Nginx
      1. No SSL needed
    6. Jira app
      1. No SSL needed
      2. BaseURL is https://jira.example.org

 

 

 

cloudfront-s-behav.PNG

 

 

CCM

Mohammed Amine Community Leader Jul 05, 2018

Thank you vm for sharing all these info. 

I'm wondering what are you using as a 'proxyName' in the 'server.xml'. Are you using jira.example.org? or jira-elb.example.org? 

kind regards

Craig Castle-Mead Community Leader Jul 05, 2018

Server.xml proxyname is JIRA.example.org

Mohammed Amine Community Leader Jul 06, 2018

Thank you vm @Craig Castle-Mead. It's finally working. The key takeaway point is using the ALB. It really changed everything. 

I had to rebuild a big portion of my infrastructure as I was mainly relying on Nginx to do what I'm now doing with the ALB. 

Really thank you vm for everything. 

Are you using a WAF with your distribution? 

I have just used a template given by Amazon and found that some Jira gadgets (like "Assigned to me" gadget) is not able to display the content

'An unknown error occurred while trying to perform a search'.

After investigation I found that the SQL Injection rule was blocking the content because the JQL query is transmitted in the URI. 

best regards

Craig Castle-Mead Community Leader May 09, 2019

Hi Mohammed,

we have tested the AWS WAF with OWASP top 10 rules from the marketplace, however it quickly started getting false positives as you found. Jira tickets often contain JavaScript code snippet as the description or comment - and WAFs see this as code injection and block it. Still looking for an option that (somehow) knows what’s good code and what’s bad code. 

 

CCM

Like Liam Bailey likes this
Liam Bailey Aug 05, 2018 • edited

Thank you @Craig Castle-Mead, this worked for me and our Confluence users are definitely benefiting from the improved performance.

We had issues initially which I think was due to using a 3rd party SSO provider, and I needed to recreate the CloudFront distribution from scratch before it would work, but all good now.

I have posed this to Atlassian support as well - Do you think there are other Confluence directories which would benefit from being served from CloudFront? E.g. is the /synchrony/* directory serving static JS content?

Thanks, Liam

Craig Castle-Mead Community Leader Aug 06, 2018

Glad to hear you saw some improvement. 

Re: other items that can be cached, I haven't dug too much further with Confluence yet (the benefits for the work done so far were quite high, so priority wise it dropped down the list), but my method is using the network panel in Chrome (or similar) to look for repeating URL patterns returning the same content, doing some digging online to try and validate my understanding of those objects - what they're used for/when they change/etc, determine what a suitable TTL would be (eg: for project avatars in Jira, if an image is cached a bit too heavily, it might be slightly annoying/confusing to a user but not a show stopper), then test it out in UAT, validate and roll to prod if happy.

/synchrony/ (based on my understanding), is all about the collaborative live editing which is unlikely to suit caching as the data is dynamic. I did however notice that /synchrony-proxy/resources/js/synchrony.min.js is a 166KB JS file without URL parameters etc, seems to be loaded, consistent file size (according to our nginx logs), requested ~ 150 times/hour in our environment, so would be a minimal saving, but every little bit helps.

 

CCM

Like Liam Bailey likes this
Neil Gladwin Sep 01, 2018

Thanks for your guide. I successfully implemented it and Jira is working fine but nothing is being cached. Every single resource is 'Miss from cloudfront', is anything under /s/* ever cached?

Craig Castle-Mead Community Leader Sep 09, 2018

Hey Neil,

The way I start testing is finding a call for a /s/ item, opening it in a new tab and refreshing a few times and monitoring the Header x-cache and see if it changes from Miss to Hit. I also added the x-cache header as a custom column in the network panel, makes it easy to see at a glance how many items are getting hits.

If a single request from /s/ never changes from Miss to Hit, revalidate the CloudFront Behaviors.

CCM

Neil Gladwin Sep 18, 2018 • edited Sep 19, 2018

Hi Craig, thanks for getting back to me. Looking at the assets under /s/* they all have "Expires" as the time it was requested. I can't see any documentation around changing this, is this something you changed or override on yours either at apache / tomcat conf or CDN?

I've also removed this at the apache level and it's still not cached. I also changed the object caching to custom and set it to 86400, still no caching. I believe it's because each load it sets a new cookie, even for just the JS files.

Craig Castle-Mead Community Leader Sep 19, 2018

Hey Neil,

Without digging deeper, I'm out of suggestions at the moment. Given there's no private messaging on here, do you want to find me on LinkedIn (I have a unique name so should be easy to find). Drop me a message and happy to see if we can line up a quick chat to go through it.

 

CCM

Yogesh Khandelwal Nov 04, 2018

I am also facing the same issue. Can we sync up on linkedin?

Yogesh Khandelwal Oct 29, 2018

Hi @Craig Castle-Mead I have enabled the same settings as yours but i am not seeing any hits in monitoring to cloudfronts. I have cloudfront then ALB then nginx and app server running. All are on 443.

Can you help me to find out what could be the cause?

Neil Gladwin Nov 14, 2018 • edited

So I believe I've found the issue. Jira sets an Expires header to the current time, hence it is always expired and will never get a hit unless done on the same second.

Unsetting it in the Apache or Nginx config will then allow it to be cached however this makes most requests served from local memory, only when someone else hits the Jira server is it served by CloudFront (and from then on local). This does make the page very fast to load comparatively. I want to spend more time on this to make it better however it just seems very hacky.

I don't like the idea of this personally because if there is an update, invalidating the CloudFront cache won't achieve anything.

Personally I think this is very bad engineering. I can understand having the need for some assets to be requested each time but from what I can tell, in Jira at least, a blanket rule across all assets, static or otherwise, to set to expire immediately.

Yogesh Khandelwal Nov 18, 2018

Thanks Neil. Can you please let me know what settings i need to unset in apace or nginx? i have ALB as well on top of apache.

Neil Gladwin Nov 19, 2018 • edited

Just unset Expires

I'd like to investigate improving the headers but I can't give a timeframe for it.

Yogesh Khandelwal Nov 23, 2018

we don't have any header set in apache but still we are getting the issue of nothing being cached.

Yogesh Khandelwal Feb 11, 2019

@Neil GladwinWe have removed the apache now and directly using ALB which is pointing to jira instances.

We are still seeing missed from cloudfront for all the requests.

2019-02-11_21-17-24.png2019-02-11_21-09-08.png2019-02-11_21-08-37.png2019-02-11_21-08-05.png2019-02-11_21-07-25.png2019-02-11_21-05-56.png2019-02-11_21-01-45.png2019-02-11_20-59-03.png

Zhenis Beisekov Atlassian Team Nov 14, 2018

Hi @Craig Castle-Mead
Thank you very much for sharing your experience. 

Could you please provide a little bit more input on:
- what locations are your users located? where is the server?
- what is the latency between different locations and the server?
- what apps/plugins do you have installed?
- how many users are concurrently online?

I ask, because I would like to figure out how more of Atlassian users can benefit.

Craig Castle-Mead Community Leader Nov 28, 2018

Hey Zhenis - sorry for the delay, I've been busy/travelling - believe Matt S. has scheduled a call next Tuesday with yourself and Jakub K. to discuss this during our TAM workshop.

 

CCM

David Barrett Jan 29, 2019

Hi @Craig Castle-Mead, many thanks for your post - very handy.

Today I think I found out why some of us were having trouble with CloudFront not caching our content. If you use CloudFront --> ALB --> JIRA, and have stickiness turned on in the target group then objects won't cache. Turning it off meant the AWSALB and Expires cookies were no longer being set, and caching started working.

It would be interesting to see if others find this useful, and if there is any way to have stickiness turned on without it breaking things.

- Dab

Like Craig Castle-Mead likes this
Craig Castle-Mead Community Leader Jan 29, 2019

Hey David,

Thats definitely something I’ll check in to on our side and report back. When I originally wrote this we were on JIRA server and ELBs. We’ve since moved to DC and ALBs (with stickysessions), will check in to what’s being cached and report back for all.

 

CCM

Craig Castle-Mead Community Leader Jan 31, 2019

Hey all,

I spent some time testing this morning and can reproduce this in our environment as well (and, not surprisingly, our cache effectiveness has dropped significantly since we moved to ALB with stickyness on).

With stickyness on (as required) - all behaviors with forced cache min/max values are being cached as expected, however the /s/* behavior which has stickyness on does not, and does not have the set-cookie header at all. With stickyness on, set-cookie exists with an Expires= value as you mentioned, and the object starts being cached.

I've lodged a request with our AWS support team and will see what they come back with and test any suggestions in our UAT enviro.

CCM

Yogesh Khandelwal Feb 08, 2019

Hi @Craig Castle-Meaddid you get any update from /s/* behavior?

Yogesh Khandelwal Feb 11, 2019

@Craig Castle-Meadall behaviors with forced cache min/max values are being cached as expected - Does this mean setting the minimum TTL to 0? or can you share your configuration?

Craig Castle-Mead Community Leader Feb 13, 2019

Working on the /s/* issue still with Stickyness on. I've yet to get a helpful reply on how to get cookie based stickyness working in harmony when the application cookies are needed by the CDN. I'm testing a few workaround configs to force caching regardless - will report back.

CCM

David Yu Feb 27, 2019

Are you turning on stickyness due to Data Center Edition? What about this dark feature (jira.fixed.cdn.enabled), would that help in routing users to a different ALB that isn't sticky?

Yogesh Khandelwal Apr 03, 2019

@Craig Castle-Meadany luck with force caching?

Craig Castle-Mead Community Leader May 09, 2019

As you may have heard, Atlassian announced CDN support for jira/confluence/Bitbucket (web UI, SSH not suported) at Summit 2019.

 

The CDN snippet from https://community.atlassian.com/t5/Removed-Posts/Sneak-peek-Improvements-coming-to-Server-and-Data-Center/td-p/1067940 is:

  • CDN support: To support better performance for distributed teams, we are introducing content delivery network (CDN) support for our core Data Center offerings. By enabling CDN support, you can accelerate the experience of your remote users as they interact with the UI of our products and reduce peak load on your primary application instances. You can use any CDN you want or a reverse proxy setup. We'll also provide easy integration with AWS CloudFront in updated quick start templates.

 

 

I’m very much looking forward to seeing how Atlassian implement the CDN setup and being able to offload our /s/ content again.

 

https://confluence.atlassian.com/adminjira/preparing-for-jira-8-2-968675818.html has a note about their underlying change to help with the support:

In Jira 7.0 we moved to stateless delivery of JavaScript and CSS resources. Apps that don't use the new APIs for web-resource transforms and conditions may cause static assets to be cached incorrectly, when CDN is enabled. See Stateless web-resource transforms and conditions to make sure your app is using the new APIs for web-resource transforms and conditions. 

We will also provide a health check on the CDN administration screen that will indicate if an app is using deprecated methods. 

These changes will not be included in Jira 8.2, but we're planning to add them in one of the next releases.

 

Ill be keeping an eye on the release notes for the first CDN version and get to testing ASAP.

 

CCM

Like # people like this
Neil Gladwin May 09, 2019

Thanks Craig. I'm a bit lost though (I don't follow the Jira updates as well as I should). Jira 7.0 is not the current release, but they moved to stateless js / css. In theory anyone with 7.x should have been able to get a CDN working if they were self hosted. But I had major issues getting it working. Though I finally did it wasn't worth the tradeoff.

Now we're currently on 8.1, but they are removing stateless on 8.2, but then adding it back in the future? This is confusing as hell.

Craig Castle-Mead Community Leader May 09, 2019

Hey @Neil Gladwin  - where would the fun (and sleepless nights) be if it was all consistent and 100% clear and unambiguous documentation :)

CCM

Like Neil Gladwin likes this

Comment

Log in or Sign up to comment
Community showcase
Published in Jira

Try Jira Cloud for Outlook: Organize your work without leaving your inbox

Hi Atlassian community, My name is Max and I work on the product integration team at Atlassian. I am pleased to announce the early access program for the Jira Cloud add-in for Outlook. This add-in...

1,636 views 3 15
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you