It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage
Highlighted

CDN/CloudFront and Atlassian Server apps

There's a few discussions around speeding up Jira/Confluence etc server installs, so thought it would be beneficial to kick start a discussion and share some learnings, and see what others had tried with what results.

The install the team I'm part of look after is for a global organization with around 15,000 users spread all around the world. We're running Server for both Jira and Confluence and to help get the most out of these and the best performance possible for our users, we looked at how we could use CDN to boost performance.

 

** disclaimer - below is based on my experience and circumstances only, if you do choose to try anything test in a non-production environment first and no warranty express or implied ** 

 

Given we're not in control of the application code itself, there was some trial and error on our dev environment, but have been able to get significant benefit from our setup.

Stats since Jan 1st, 2018:

Jira: Of the 689GB traffic initiated by users browsers, 587GB has been served directly from CloudFront, leaving only 102GB needing to go back to the server (origin) ~ 85%

Confluence: Of 185GB total, 160GB has come from CloudFront, 26GB from the server ~ 86%

 

While the traffic that does make it back to the server is where the server needs to do most work (dynamic content), offloading such a large percentage of the easy traffic has allowed us to get more life out of the infrastructure and licensing we do have. 

 

NB: Even if you choose to enable no caching whatsoever, enabling a CDN will (should) assist with performance as end users traffic will go from their ISP to the AWS backbone at the closest edge location and then travel along the AWS backbone to your server. 

 

Given most of the objects that are requested from the server require authentication, you'll need to pass through cookies. There's also heavy use of URI queries/params, so these are passed through as well.

When doing an app or add-on update, likely safest to run a full cache invalidation once complete to ensure no old versions are being served.

Anything not mentioned is using the default setting:

JIRA

  1. Path Pattern = /s/*
    1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Use Origin Cache Headers
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, cache based on all
  2. Path pattern = /secure/projectavatar*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 86400
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. pid
        size
  3. Path pattern = /secure/viewavatar*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 86400
    4. Forward Cookies = All
    5. Query String Forwarding and Caching = Forward all, based on whitelist
      1. avatarId
        avatarType
        size
  4. Path pattern = /download/resources/*
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Default = 86400
      2. Max = 31536000
    4. Forward Cookies = None
    5. Query String Forwarding and Caching = Forward all, cache based on all
  5. Path pattern = /robots.txt and /favicon.ico (cause hey, why not)
    1. Allowed HTTP methods = GET, HEAD
    2. Cache Based on Selected Request Headers = Whitelist
    3. Object Caching = Customize
      1. Min/Max/Default = 31536000
    4. Forward Cookies = None
    5. Query String Forwarding and Caching = None
  6. Default:
    1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE  
    2. Cache Based on Selected Request Headers = All
    3. Forward Cookies = All
    4. Query String Forwarding and Caching = Forward all, cache based on all

 

Confluence:

We're just doing the same /s/* for Confluence for now.

 

If anyone has any feedback, it'd be great to hear how you were able to improve performance. Or, even if you tried something but it didn't work.

 

CCM

 

8 comments

Hi, I am trying to do the same to get better access speed for Jira. What dud you use for CDN and how are you hosting your instances

Hi @Itesh Shah

We're using AWS CloudFront as the CDN.

What do you mean by your "how are you hosting your instances?" They're a combination of M4 and M5 EC2 instances in AWS eu-central-1 - this made most sense from a compliance and central geographical point of view

 

CCM

Hi,

Thank you for the revert to my query.

How are you hosting your instances I meant wether they are "On-prem in a datacenter" or "Hosting in the cloud (AWS/Google/etc) sorry that I did not clarify what i meant. Thank you though for letting me know that you are doing it in AWS.


We are having an "On-prem" situation and trying to use cloud front.
Any special changes that needs to be done on Jira other than enabling the dark feature? and any special attention that I need to pay for CloudFront configs?

 

Regards,

Itesh

Hi Itesh,

Sorry for the belated reply - totally missed the notification.

We did not change anything in the Jira config at all - it was just a bunch of network inspection in browsers and looking for URL patterns that returned consistent content and mapping these to behaviors with varying levels of caching. The default behavior enforces no cache so just sends the entire request (all headers, cookies, query strings, methods) back to the origin as if the CDN didn't exist.

CCM

This is a very informative list, but I'm a little stuck as to what headers you are whitelisting in the caching settings.

To answer my own question, whitelisting "Origin" headers seems to do most of the trick, but I'm having issues with woff and ttf files for some reason. They end up cancelling entirely.

Hey Robert,

Not sure what could be happening with your ttf/woff's - maybe worth putting a rule just for *.woff and *.ttf at the top of your rules and forcing them through without any cache to check if they at least load, and if that works, you can change the behavior so they then should be cached - at least this way you know exactly which rule you're diagnosing (actually, it may be worth adding a custom header to each of your behaviors (rule = 1, rule = 2 etc etc) so that you can view the headers of each file and it will tell you which rule the file is using)

CCM

Digging in a little more on the .woff and .ttf, they're transferring correctly in some browsers, but CloudFront is returning a 502 in others, even if I set them up in a passthrough rule.

For the browsers where they do transfer, they fail to load/display due to CORS, seeing as how their Origin is my Jira instance but they're being downloaded through cloudfront.net. I'm currently working that angle, but have been unable to set Access-Control-Allow-Origin headers on the requests.

Mohammed Amine Community Leader Jul 04, 2018

Thank you vm @Craig Castle-Mead for the post. 

Tried to do the samething but I'm wondering how would I disable any traffic not getting from the cloudfront distribution? 

In my case I'm using an EC2 instance with a reverse-proxy sending traffic to Jira & Confluence. 

thanks

MKA

Hey Mohammed,

To ensure only CloudFront traffic is getting through to your environment, you'll want to have the security group automatically update based on the IP ranges CloudFront uses. As these can (and do) change, AWS have an SNS topic you can subscribe, this can then trigger a Lambda function that updates the security group inbound rules (write up at https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/ )

PS. Even if you only have a single EC2 instance, I'd suggest putting a load balancer in front it and only allowing 80/443 to the load balancer from CloudFront and then only allowing 80/443 to the EC2 instance from the load balancer.

 

CCM

Mohammed Amine Community Leader Jul 04, 2018

Thank you for your reply @Craig Castle-Mead

Let me give you come more details about the infrastructure I'm managing:

  • Cloudfront
  • Origin: EC2 Instance in a public Subnet
    • NACL & SG (allowing only 80/443) + will add the lambda function (thanks ;) )
    • Nginx (reverse Proxy): forwarding 80(http)->443(https) + SSL + ... etc
    • Crowd
    • Jira
    • Confluence
    • ...
  • RDS on a private Subnet

Do you think that I need to add the LB?

Besides, I'm still not able to deploy the CDN. I went through all the steps here but still getting 502 error page. I'm missing something that I cann't figure out !


Here is my configuration:

 

Capture d’écran 2018-07-04 à 17.17.50.pngCNAME: I'm using 'xxx.domainname.com' for which I have created a CNAME record in my host (OVH).

Capture d’écran 2018-07-04 à 17.18.01.pngCapture d’écran 2018-07-04 à 17.18.52.png

I have a generated a SSL certificate that I have uploaded to ACM and used in cloudfront. 

While troubleshooting I have done the following:

  • Used an SSL checker (https://www.sslchecker.com/sslchecker) which tells me that the certificate and the chain were find but not the root. BTW other online SSL checkers say that everything is ok. Don't know if the issue is coming from here. Amazon CM finds that the private, certificate and the chain are OK. 
  • Used Openssl
    • openssl s_client –connect domainname:443 –servername domainname ==> is OK
    • openssl s_client –connect domainname:443 ==> is giving me SSL HANDSHAKE ERROR. But Cloudfront is not configured to allow non SNI clients. So I think it is ok
  • Checked my domaine with https://www.ssllabs.com/ssltest/ for ciphers & protocols. Everything seem to be ok. I can give more details if necessary

Really cann't find what I'm missing !!

Thank you for your help

kind regards

Just to follow up and share some information that i've just got from Amazon support: the ssl check I did was apparently the wrong one. I had to use the EC2 custom domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com)and not my custom domain name (xxx.mydomainname.com).

So I'm now wondering if I have to generate a certificate for the ec2 domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com) and use that for both cloudfront and my Nginx Server (installed in my EC2 instance)? and just delete (and forget about) my custom domain name SSL certificate?

And what shall I use as base URL for Jira and confluence?

@Craig Castle-Mead for the "Path Pattern = /s/*", you said that you used "Cache Based on Selected Request Headers = Whitelist". Can you please share the whitelist? and also share what you are using as a base URL for Jira (is it your domain name of the ec2 custom domain name)

thank you vm

Mohammed Amine Community Leader Jul 05, 2018

Just for followup, it is impossible to generate SSL certificate using letsencrypt for ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com !!

Hey Mohammed,

  1. /s/* behaviour screenshot below
  2. If we assume your end users are going to access jira from https://jira.example.org
    1. Get a wildcard SSL for example.org
    2. Deploy using AWS Certificate Manager
    3. CloudFront
      1. Use the wildcard cert from Cert Manager
      2. CNAME should be jira.example.org
      3. Origin should be the CNAME of the ELB (eg: jira-example-org-123456.us-west-1.elb.amazonaws.com)
    4. ELB
      1. Use the wildcard cert from Cert Manager
      2. Point to HTTPS listener to HTTP port on the EC2 instances (SSL offloading)
      3. We also CNAME jira-elb.example.org to the ELB. We've had SSL issues when using BobSwift CLI going through CloudFront but when pointing BobSwift to the -elb hostname all works OK
    5. Nginx
      1. No SSL needed
    6. Jira app
      1. No SSL needed
      2. BaseURL is https://jira.example.org

 

 

 

cloudfront-s-behav.PNG

 

 

CCM

Mohammed Amine Community Leader Jul 05, 2018

Thank you vm for sharing all these info. 

I'm wondering what are you using as a 'proxyName' in the 'server.xml'. Are you using jira.example.org? or jira-elb.example.org? 

kind regards

Server.xml proxyname is JIRA.example.org

Mohammed Amine Community Leader Jul 06, 2018

Thank you vm @Craig Castle-Mead. It's finally working. The key takeaway point is using the ALB. It really changed everything. 

I had to rebuild a big portion of my infrastructure as I was mainly relying on Nginx to do what I'm now doing with the ALB. 

Really thank you vm for everything. 

Are you using a WAF with your distribution? 

I have just used a template given by Amazon and found that some Jira gadgets (like "Assigned to me" gadget) is not able to display the content

'An unknown error occurred while trying to perform a search'.

After investigation I found that the SQL Injection rule was blocking the content because the JQL query is transmitted in the URI. 

best regards

Thank you @Craig Castle-Mead, this worked for me and our Confluence users are definitely benefiting from the improved performance.

We had issues initially which I think was due to using a 3rd party SSO provider, and I needed to recreate the CloudFront distribution from scratch before it would work, but all good now.

I have posed this to Atlassian support as well - Do you think there are other Confluence directories which would benefit from being served from CloudFront? E.g. is the /synchrony/* directory serving static JS content?

Thanks, Liam

Glad to hear you saw some improvement. 

Re: other items that can be cached, I haven't dug too much further with Confluence yet (the benefits for the work done so far were quite high, so priority wise it dropped down the list), but my method is using the network panel in Chrome (or similar) to look for repeating URL patterns returning the same content, doing some digging online to try and validate my understanding of those objects - what they're used for/when they change/etc, determine what a suitable TTL would be (eg: for project avatars in Jira, if an image is cached a bit too heavily, it might be slightly annoying/confusing to a user but not a show stopper), then test it out in UAT, validate and roll to prod if happy.

/synchrony/ (based on my understanding), is all about the collaborative live editing which is unlikely to suit caching as the data is dynamic. I did however notice that /synchrony-proxy/resources/js/synchrony.min.js is a 166KB JS file without URL parameters etc, seems to be loaded, consistent file size (according to our nginx logs), requested ~ 150 times/hour in our environment, so would be a minimal saving, but every little bit helps.

 

CCM

Like 1 person likes this

Thanks for your guide. I successfully implemented it and Jira is working fine but nothing is being cached. Every single resource is 'Miss from cloudfront', is anything under /s/* ever cached?

Hey Neil,

The way I start testing is finding a call for a /s/ item, opening it in a new tab and refreshing a few times and monitoring the Header x-cache and see if it changes from Miss to Hit. I also added the x-cache header as a custom column in the network panel, makes it easy to see at a glance how many items are getting hits.

If a single request from /s/ never changes from Miss to Hit, revalidate the CloudFront Behaviors.

CCM

Hi Craig, thanks for getting back to me. Looking at the assets under /s/* they all have "Expires" as the time it was requested. I can't see any documentation around changing this, is this something you changed or override on yours either at apache / tomcat conf or CDN?

I've also removed this at the apache level and it's still not cached. I also changed the object caching to custom and set it to 86400, still no caching. I believe it's because each load it sets a new cookie, even for just the JS files.

Hey Neil,

Without digging deeper, I'm out of suggestions at the moment. Given there's no private messaging on here, do you want to find me on LinkedIn (I have a unique name so should be easy to find). Drop me a message and happy to see if we can line up a quick chat to go through it.

 

CCM

I am also facing the same issue. Can we sync up on linkedin?

Hi @Craig Castle-Mead I have enabled the same settings as yours but i am not seeing any hits in monitoring to cloudfronts. I have cloudfront then ALB then nginx and app server running. All are on 443.

Can you help me to find out what could be the cause?

So I believe I've found the issue. Jira sets an Expires header to the current time, hence it is always expired and will never get a hit unless done on the same second.

Unsetting it in the Apache or Nginx config will then allow it to be cached however this makes most requests served from local memory, only when someone else hits the Jira server is it served by CloudFront (and from then on local). This does make the page very fast to load comparatively. I want to spend more time on this to make it better however it just seems very hacky.

I don't like the idea of this personally because if there is an update, invalidating the CloudFront cache won't achieve anything.

Personally I think this is very bad engineering. I can understand having the need for some assets to be requested each time but from what I can tell, in Jira at least, a blanket rule across all assets, static or otherwise, to set to expire immediately.

Thanks Neil. Can you please let me know what settings i need to unset in apace or nginx? i have ALB as well on top of apache.

Just unset Expires

I'd like to investigate improving the headers but I can't give a timeframe for it.

we don't have any header set in apache but still we are getting the issue of nothing being cached.

@Neil GladwinWe have removed the apache now and directly using ALB which is pointing to jira instances.

We are still seeing missed from cloudfront for all the requests.

2019-02-11_21-17-24.png2019-02-11_21-09-08.png2019-02-11_21-08-37.png2019-02-11_21-08-05.png2019-02-11_21-07-25.png2019-02-11_21-05-56.png2019-02-11_21-01-45.png2019-02-11_20-59-03.png

Hi @Craig Castle-Mead
Thank you very much for sharing your experience. 

Could you please provide a little bit more input on:
- what locations are your users located? where is the server?
- what is the latency between different locations and the server?
- what apps/plugins do you have installed?
- how many users are concurrently online?

I ask, because I would like to figure out how more of Atlassian users can benefit.

Hey Zhenis - sorry for the delay, I've been busy/travelling - believe Matt S. has scheduled a call next Tuesday with yourself and Jakub K. to discuss this during our TAM workshop.

 

CCM

Hi @Craig Castle-Mead, many thanks for your post - very handy.

Today I think I found out why some of us were having trouble with CloudFront not caching our content. If you use CloudFront --> ALB --> JIRA, and have stickiness turned on in the target group then objects won't cache. Turning it off meant the AWSALB and Expires cookies were no longer being set, and caching started working.

It would be interesting to see if others find this useful, and if there is any way to have stickiness turned on without it breaking things.

- Dab

Like 1 person likes this

Hey David,

Thats definitely something I’ll check in to on our side and report back. When I originally wrote this we were on JIRA server and ELBs. We’ve since moved to DC and ALBs (with stickysessions), will check in to what’s being cached and report back for all.

 

CCM

Hey all,

I spent some time testing this morning and can reproduce this in our environment as well (and, not surprisingly, our cache effectiveness has dropped significantly since we moved to ALB with stickyness on).

With stickyness on (as required) - all behaviors with forced cache min/max values are being cached as expected, however the /s/* behavior which has stickyness on does not, and does not have the set-cookie header at all. With stickyness on, set-cookie exists with an Expires= value as you mentioned, and the object starts being cached.

I've lodged a request with our AWS support team and will see what they come back with and test any suggestions in our UAT enviro.

CCM

Hi @Craig Castle-Meaddid you get any update from /s/* behavior?

@Craig Castle-Meadall behaviors with forced cache min/max values are being cached as expected - Does this mean setting the minimum TTL to 0? or can you share your configuration?

Working on the /s/* issue still with Stickyness on. I've yet to get a helpful reply on how to get cookie based stickyness working in harmony when the application cookies are needed by the CDN. I'm testing a few workaround configs to force caching regardless - will report back.

CCM

Are you turning on stickyness due to Data Center Edition? What about this dark feature (jira.fixed.cdn.enabled), would that help in routing users to a different ALB that isn't sticky?

Comment

Log in or Sign up to comment
Community showcase
Published Apr 09, 2019 in Portfolio for Jira

Portfolio for Jira 3.0 is here!

The wait is over... Portfolio for Jira Server and Data Center 3.0 is now officially here! Platform releases offer Atlassian an opportunity to shift our strategy, make bold predictions about t...

1,522 views 14 26
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you