nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

CDN/CloudFront and Atlassian Server apps

Craig Castle-Mead_20fp9z

There's a few discussions around speeding up Jira/Confluence etc server installs, so thought it would be beneficial to kick start a discussion and share some learnings, and see what others had tried with what results.

The install the team I'm part of look after is for a global organization with around 15,000 users spread all around the world. We're running Server for both Jira and Confluence and to help get the most out of these and the best performance possible for our users, we looked at how we could use CDN to boost performance.

** disclaimer - below is based on my experience and circumstances only, if you do choose to try anything test in a non-production environment first and no warranty express or implied **

Given we're not in control of the application code itself, there was some trial and error on our dev environment, but have been able to get significant benefit from our setup.

Stats since Jan 1st, 2018:

Jira: Of the 689GB traffic initiated by users browsers, 587GB has been served directly from CloudFront, leaving only 102GB needing to go back to the server (origin) ~ 85%

Confluence: Of 185GB total, 160GB has come from CloudFront, 26GB from the server ~ 86%

While the traffic that does make it back to the server is where the server needs to do most work (dynamic content), offloading such a large percentage of the easy traffic has allowed us to get more life out of the infrastructure and licensing we do have.

NB: Even if you choose to enable no caching whatsoever, enabling a CDN will (should) assist with performance as end users traffic will go from their ISP to the AWS backbone at the closest edge location and then travel along the AWS backbone to your server.

Given most of the objects that are requested from the server require authentication, you'll need to pass through cookies. There's also heavy use of URI queries/params, so these are passed through as well.

When doing an app or add-on update, likely safest to run a full cache invalidation once complete to ensure no old versions are being served.

Anything not mentioned is using the default setting:

JIRA

Path Pattern = /s/*
1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
2. Cache Based on Selected Request Headers = Whitelist
3. Object Caching = Use Origin Cache Headers
4. Forward Cookies = All
5. Query String Forwarding and Caching = Forward all, cache based on all
Path pattern = /secure/projectavatar*
1. Allowed HTTP methods = GET, HEAD
2. Cache Based on Selected Request Headers = Whitelist
3. Object Caching = Customize
  1. Min/Max/Default = 86400
4. Forward Cookies = All
5. Query String Forwarding and Caching = Forward all, based on whitelist
  1. pid
    size
Path pattern = /secure/viewavatar*
1. Allowed HTTP methods = GET, HEAD
2. Cache Based on Selected Request Headers = Whitelist
3. Object Caching = Customize
  1. Min/Max/Default = 86400
4. Forward Cookies = All
5. Query String Forwarding and Caching = Forward all, based on whitelist
  1. avatarId
    avatarType
    size
Path pattern = /download/resources/*
1. Allowed HTTP methods = GET, HEAD
2. Cache Based on Selected Request Headers = Whitelist
3. Object Caching = Customize
  1. Min/Default = 86400
  2. Max = 31536000
4. Forward Cookies = None
5. Query String Forwarding and Caching = Forward all, cache based on all
Path pattern = /robots.txt and /favicon.ico (cause hey, why not)
1. Allowed HTTP methods = GET, HEAD
2. Cache Based on Selected Request Headers = Whitelist
3. Object Caching = Customize
  1. Min/Max/Default = 31536000
4. Forward Cookies = None
5. Query String Forwarding and Caching = None
Default:
1. Allowed HTTP methods = GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
2. Cache Based on Selected Request Headers = All
3. Forward Cookies = All
4. Query String Forwarding and Caching = Forward all, cache based on all

Confluence:

We're just doing the same /s/* for Confluence for now.

If anyone has any feedback, it'd be great to hear how you were able to improve performance. Or, even if you tried something but it didn't work.

CCM

Accepted answers

All comments

Itesh Shah

Hi, I am trying to do the same to get better access speed for Jira. What dud you use for CDN and how are you hosting your instances

Craig Castle-Mead_20fp9z

Hi @Itesh Shah

We're using AWS CloudFront as the CDN.

What do you mean by your "how are you hosting your instances?" They're a combination of M4 and M5 EC2 instances in AWS eu-central-1 - this made most sense from a compliance and central geographical point of view

CCM

Itesh Shah

Hi,

Thank you for the revert to my query.

How are you hosting your instances I meant wether they are "On-prem in a datacenter" or "Hosting in the cloud (AWS/Google/etc) sorry that I did not clarify what i meant. Thank you though for letting me know that you are doing it in AWS.

We are having an "On-prem" situation and trying to use cloud front.
Any special changes that needs to be done on Jira other than enabling the dark feature? and any special attention that I need to pay for CloudFront configs?

Regards,

Itesh

Robert Beckman

This is a very informative list, but I'm a little stuck as to what headers you are whitelisting in the caching settings.

Craig Castle-Mead_20fp9z

Hi Itesh,

Sorry for the belated reply - totally missed the notification.

We did not change anything in the Jira config at all - it was just a bunch of network inspection in browsers and looking for URL patterns that returned consistent content and mapping these to behaviors with varying levels of caching. The default behavior enforces no cache so just sends the entire request (all headers, cookies, query strings, methods) back to the origin as if the CDN didn't exist.

CCM

Robert Beckman

To answer my own question, whitelisting "Origin" headers seems to do most of the trick, but I'm having issues with woff and ttf files for some reason. They end up cancelling entirely.

M Amine

Thank you vm @Craig Castle-Mead for the post.

Tried to do the samething but I'm wondering how would I disable any traffic not getting from the cloudfront distribution?

In my case I'm using an EC2 instance with a reverse-proxy sending traffic to Jira & Confluence.

thanks

MKA

Craig Castle-Mead_20fp9z

Hey Mohammed,

To ensure only CloudFront traffic is getting through to your environment, you'll want to have the security group automatically update based on the IP ranges CloudFront uses. As these can (and do) change, AWS have an SNS topic you can subscribe, this can then trigger a Lambda function that updates the security group inbound rules (write up at https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/ )

PS. Even if you only have a single EC2 instance, I'd suggest putting a load balancer in front it and only allowing 80/443 to the load balancer from CloudFront and then only allowing 80/443 to the EC2 instance from the load balancer.

CCM

Craig Castle-Mead_20fp9z

Hey Robert,

Not sure what could be happening with your ttf/woff's - maybe worth putting a rule just for *.woff and *.ttf at the top of your rules and forcing them through without any cache to check if they at least load, and if that works, you can change the behavior so they then should be cached - at least this way you know exactly which rule you're diagnosing (actually, it may be worth adding a custom header to each of your behaviors (rule = 1, rule = 2 etc etc) so that you can view the headers of each file and it will tell you which rule the file is using)

CCM

M Amine

Thank you for your reply @Craig Castle-Mead.

Let me give you come more details about the infrastructure I'm managing:

Cloudfront
Origin: EC2 Instance in a public Subnet
- NACL & SG (allowing only 80/443) + will add the lambda function (thanks )
- Nginx (reverse Proxy): forwarding 80(http)->443(https) + SSL + ... etc
- Crowd
- Jira
- Confluence
- ...
RDS on a private Subnet

Do you think that I need to add the LB?

Besides, I'm still not able to deploy the CDN. I went through all the steps here but still getting 502 error page. I'm missing something that I cann't figure out !

Here is my configuration:

Capture d’écran 2018-07-04 à 17.17.50.png CNAME: I'm using 'xxx.domainname.com' for which I have created a CNAME record in my host (OVH).

Capture d’écran 2018-07-04 à 17.18.01.png

I have a generated a SSL certificate that I have uploaded to ACM and used in cloudfront.

While troubleshooting I have done the following:

Used an SSL checker (https://www.sslchecker.com/sslchecker) which tells me that the certificate and the chain were find but not the root. BTW other online SSL checkers say that everything is ok. Don't know if the issue is coming from here. Amazon CM finds that the private, certificate and the chain are OK.
Used Openssl
- openssl s_client –connect domainname:443 –servername domainname ==> is OK
- openssl s_client –connect domainname:443 ==> is giving me SSL HANDSHAKE ERROR. But Cloudfront is not configured to allow non SNI clients. So I think it is ok
Checked my domaine with https://www.ssllabs.com/ssltest/ for ciphers & protocols. Everything seem to be ok. I can give more details if necessary

Really cann't find what I'm missing !!

Thank you for your help

kind regards

M Amine

Just to follow up and share some information that i've just got from Amazon support: the ssl check I did was apparently the wrong one. I had to use the EC2 custom domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com)and not my custom domain name (xxx.mydomainname.com).

So I'm now wondering if I have to generate a certificate for the ec2 domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com) and use that for both cloudfront and my Nginx Server (installed in my EC2 instance)? and just delete (and forget about) my custom domain name SSL certificate?

And what shall I use as base URL for Jira and confluence?

@Craig Castle-Mead for the "Path Pattern = /s/*", you said that you used "Cache Based on Selected Request Headers = Whitelist". Can you please share the whitelist? and also share what you are using as a base URL for Jira (is it your domain name of the ec2 custom domain name)

thank you vm

M Amine

Just for followup, it is impossible to generate SSL certificate using letsencrypt for ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com !!

Craig Castle-Mead_20fp9z

Hey Mohammed,

/s/* behaviour screenshot below
If we assume your end users are going to access jira from https://jira.example.org
1. Get a wildcard SSL for example.org
2. Deploy using AWS Certificate Manager
3. CloudFront
  1. Use the wildcard cert from Cert Manager
  2. CNAME should be jira.example.org
  3. Origin should be the CNAME of the ELB (eg: jira-example-org-123456.us-west-1.elb.amazonaws.com)
4. ELB
  1. Use the wildcard cert from Cert Manager
  2. Point to HTTPS listener to HTTP port on the EC2 instances (SSL offloading)
  3. We also CNAME jira-elb.example.org to the ELB. We've had SSL issues when using BobSwift CLI going through CloudFront but when pointing BobSwift to the -elb hostname all works OK
5. Nginx
  1. No SSL needed
6. Jira app
  1. No SSL needed
  2. BaseURL is https://jira.example.org

CCM

M Amine

Thank you vm for sharing all these info.

I'm wondering what are you using as a 'proxyName' in the 'server.xml'. Are you using jira.example.org? or jira-elb.example.org?

kind regards

Craig Castle-Mead_20fp9z

Server.xml proxyname is JIRA.example.org

M Amine

Thank you vm @Craig Castle-Mead. It's finally working. The key takeaway point is using the ALB. It really changed everything.

I had to rebuild a big portion of my infrastructure as I was mainly relying on Nginx to do what I'm now doing with the ALB.

Really thank you vm for everything.

Are you using a WAF with your distribution?

I have just used a template given by Amazon and found that some Jira gadgets (like "Assigned to me" gadget) is not able to display the content

'An unknown error occurred while trying to perform a search'.

After investigation I found that the SQL Injection rule was blocking the content because the JQL query is transmitted in the URI.

best regards

Robert Beckman

Digging in a little more on the .woff and .ttf, they're transferring correctly in some browsers, but CloudFront is returning a 502 in others, even if I set them up in a passthrough rule.

For the browsers where they do transfer, they fail to load/display due to CORS, seeing as how their Origin is my Jira instance but they're being downloaded through cloudfront.net. I'm currently working that angle, but have been unable to set Access-Control-Allow-Origin headers on the requests.

lbailey79

Thank you @Craig Castle-Mead, this worked for me and our Confluence users are definitely benefiting from the improved performance.

We had issues initially which I think was due to using a 3rd party SSO provider, and I needed to recreate the CloudFront distribution from scratch before it would work, but all good now.

I have posed this to Atlassian support as well - Do you think there are other Confluence directories which would benefit from being served from CloudFront? E.g. is the /synchrony/* directory serving static JS content?

Thanks, Liam

Craig Castle-Mead_20fp9z

Glad to hear you saw some improvement.

Re: other items that can be cached, I haven't dug too much further with Confluence yet (the benefits for the work done so far were quite high, so priority wise it dropped down the list), but my method is using the network panel in Chrome (or similar) to look for repeating URL patterns returning the same content, doing some digging online to try and validate my understanding of those objects - what they're used for/when they change/etc, determine what a suitable TTL would be (eg: for project avatars in Jira, if an image is cached a bit too heavily, it might be slightly annoying/confusing to a user but not a show stopper), then test it out in UAT, validate and roll to prod if happy.

/synchrony/ (based on my understanding), is all about the collaborative live editing which is unlikely to suit caching as the data is dynamic. I did however notice that /synchrony-proxy/resources/js/synchrony.min.js is a 166KB JS file without URL parameters etc, seems to be loaded, consistent file size (according to our nginx logs), requested ~ 150 times/hour in our environment, so would be a minimal saving, but every little bit helps.

CCM

Rudiger

Thanks for your guide. I successfully implemented it and Jira is working fine but nothing is being cached. Every single resource is 'Miss from cloudfront', is anything under /s/* ever cached?

Craig Castle-Mead_20fp9z

Hey Neil,

The way I start testing is finding a call for a /s/ item, opening it in a new tab and refreshing a few times and monitoring the Header x-cache and see if it changes from Miss to Hit. I also added the x-cache header as a custom column in the network panel, makes it easy to see at a glance how many items are getting hits.

If a single request from /s/ never changes from Miss to Hit, revalidate the CloudFront Behaviors.

CCM

Rudiger

Hi Craig, thanks for getting back to me. Looking at the assets under /s/* they all have "Expires" as the time it was requested. I can't see any documentation around changing this, is this something you changed or override on yours either at apache / tomcat conf or CDN?

I've also removed this at the apache level and it's still not cached. I also changed the object caching to custom and set it to 86400, still no caching. I believe it's because each load it sets a new cookie, even for just the JS files.

Craig Castle-Mead_20fp9z

Hey Neil,

Without digging deeper, I'm out of suggestions at the moment. Given there's no private messaging on here, do you want to find me on LinkedIn (I have a unique name so should be easy to find). Drop me a message and happy to see if we can line up a quick chat to go through it.

CCM

Yogesh Khandelwal

Hi @Craig Castle-Mead I have enabled the same settings as yours but i am not seeing any hits in monitoring to cloudfronts. I have cloudfront then ALB then nginx and app server running. All are on 443.

Can you help me to find out what could be the cause?

Yogesh Khandelwal

I am also facing the same issue. Can we sync up on linkedin?

Zhenis Beisekov

Hi @Craig Castle-Mead
Thank you very much for sharing your experience.

Could you please provide a little bit more input on:
- what locations are your users located? where is the server?
- what is the latency between different locations and the server?
- what apps/plugins do you have installed?
- how many users are concurrently online?

I ask, because I would like to figure out how more of Atlassian users can benefit.

Rudiger

So I believe I've found the issue. Jira sets an Expires header to the current time, hence it is always expired and will never get a hit unless done on the same second.

Unsetting it in the Apache or Nginx config will then allow it to be cached however this makes most requests served from local memory, only when someone else hits the Jira server is it served by CloudFront (and from then on local). This does make the page very fast to load comparatively. I want to spend more time on this to make it better however it just seems very hacky.

I don't like the idea of this personally because if there is an update, invalidating the CloudFront cache won't achieve anything.

Personally I think this is very bad engineering. I can understand having the need for some assets to be requested each time but from what I can tell, in Jira at least, a blanket rule across all assets, static or otherwise, to set to expire immediately.

Yogesh Khandelwal

Thanks Neil. Can you please let me know what settings i need to unset in apace or nginx? i have ALB as well on top of apache.

Rudiger

Just unset Expires

I'd like to investigate improving the headers but I can't give a timeframe for it.

Yogesh Khandelwal

we don't have any header set in apache but still we are getting the issue of nothing being cached.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

jira

jira-service-desk