CDN/CloudFront and Atlassian Server apps

[Yogesh Khandelwal]

Thanks Neil. Can you please let me know what settings i need to unset in apace or nginx? i have ALB as well on top of apache.

[Rudiger]

So I believe I've found the issue. Jira sets an Expires header to the current time, hence it is always expired and will never get a hit unless done on the same second.

Unsetting it in the Apache or Nginx config will then allow it to be cached however this makes most requests served from local memory, only when someone else hits the Jira server is it served by CloudFront (and from then on local). This does make the page very fast to load comparatively. I want to spend more time on this to make it better however it just seems very hacky.

I don't like the idea of this personally because if there is an update, invalidating the CloudFront cache won't achieve anything.

Personally I think this is very bad engineering. I can understand having the need for some assets to be requested each time but from what I can tell, in Jira at least, a blanket rule across all assets, static or otherwise, to set to expire immediately.

[Zhenis Beisekov]

Hi @Craig Castle-Mead
Thank you very much for sharing your experience. 

Could you please provide a little bit more input on:
- what locations are your users located? where is the server?
- what is the latency between different locations and the server?
- what apps/plugins do you have installed?
- how many users are concurrently online?

I ask, because I would like to figure out how more of Atlassian users can benefit.

[Yogesh Khandelwal]

I am also facing the same issue. Can we sync up on linkedin?

[Yogesh Khandelwal]

Hi @Craig Castle-Mead I have enabled the same settings as yours but i am not seeing any hits in monitoring to cloudfronts. I have cloudfront then ALB then nginx and app server running. All are on 443.

Can you help me to find out what could be the cause?

[Craig Castle-Mead]

Hey Neil,

Without digging deeper, I'm out of suggestions at the moment. Given there's no private messaging on here, do you want to find me on LinkedIn (I have a unique name so should be easy to find). Drop me a message and happy to see if we can line up a quick chat to go through it.

CCM

[Rudiger]

Hi Craig, thanks for getting back to me. Looking at the assets under /s/* they all have "Expires" as the time it was requested. I can't see any documentation around changing this, is this something you changed or override on yours either at apache / tomcat conf or CDN?

I've also removed this at the apache level and it's still not cached. I also changed the object caching to custom and set it to 86400, still no caching. I believe it's because each load it sets a new cookie, even for just the JS files.

[Craig Castle-Mead]

Hey Neil,

The way I start testing is finding a call for a /s/ item, opening it in a new tab and refreshing a few times and monitoring the Header x-cache and see if it changes from Miss to Hit. I also added the x-cache header as a custom column in the network panel, makes it easy to see at a glance how many items are getting hits.

If a single request from /s/ never changes from Miss to Hit, revalidate the CloudFront Behaviors.

CCM

[Rudiger]

Thanks for your guide. I successfully implemented it and Jira is working fine but nothing is being cached. Every single resource is 'Miss from cloudfront', is anything under /s/* ever cached?

[lbailey79]

Thank you @Craig Castle-Mead, this worked for me and our Confluence users are definitely benefiting from the improved performance.

We had issues initially which I think was due to using a 3rd party SSO provider, and I needed to recreate the CloudFront distribution from scratch before it would work, but all good now.

I have posed this to Atlassian support as well - Do you think there are other Confluence directories which would benefit from being served from CloudFront? E.g. is the /synchrony/* directory serving static JS content?

Thanks, Liam

[Robert Beckman]

Digging in a little more on the .woff and .ttf, they're transferring correctly in some browsers, but CloudFront is returning a 502 in others, even if I set them up in a passthrough rule.

For the browsers where they do transfer, they fail to load/display due to CORS, seeing as how their Origin is my Jira instance but they're being downloaded through cloudfront.net. I'm currently working that angle, but have been unable to set Access-Control-Allow-Origin headers on the requests.

[M Amine]

Thank you vm @Craig Castle-Mead. It's finally working. The key takeaway point is using the ALB. It really changed everything. 

I had to rebuild a big portion of my infrastructure as I was mainly relying on Nginx to do what I'm now doing with the ALB. 

Really thank you vm for everything. 

Are you using a WAF with your distribution? 

I have just used a template given by Amazon and found that some Jira gadgets (like "Assigned to me" gadget) is not able to display the content

'An unknown error occurred while trying to perform a search'.

After investigation I found that the SQL Injection rule was blocking the content because the JQL query is transmitted in the URI. 

best regards

[Craig Castle-Mead]

Server.xml proxyname is JIRA.example.org

[M Amine]

Thank you vm for sharing all these info. 

I'm wondering what are you using as a 'proxyName' in the 'server.xml'. Are you using jira.example.org? or jira-elb.example.org? 

kind regards

[Craig Castle-Mead]

Hey Mohammed,

1. /s/* behaviour screenshot below
2. If we assume your end users are going to access jira from https://jira.example.org
   
   1. Get a wildcard SSL for example.org
   2. Deploy using AWS Certificate Manager
   3. CloudFront
      1. Use the wildcard cert from Cert Manager
      2. CNAME should be jira.example.org
      3. Origin should be the CNAME of the ELB (eg: jira-example-org-123456.us-west-1.elb.amazonaws.com)
   4. ELB
      1. Use the wildcard cert from Cert Manager
      2. Point to HTTPS listener to HTTP port on the EC2 instances (SSL offloading)
      3. We also CNAME jira-elb.example.org to the ELB. We've had SSL issues when using BobSwift CLI going through CloudFront but when pointing BobSwift to the -elb hostname all works OK
   5. Nginx
      1. No SSL needed
   6. Jira app
      1. No SSL needed
      2. BaseURL is https://jira.example.org

CCM

[M Amine]

Just for followup, it is impossible to generate SSL certificate using letsencrypt for ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com !!

[M Amine]

Just to follow up and share some information that i've just got from Amazon support: the ssl check I did was apparently the wrong one. I had to use the EC2 custom domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com)and not my custom domain name (xxx.mydomainname.com).

So I'm now wondering if I have to generate a certificate for the ec2 domain name (ec2-XX-XX-XXX-XXX.XX-XX-X.compute.amazonaws.com) and use that for both cloudfront and my Nginx Server (installed in my EC2 instance)? and just delete (and forget about) my custom domain name SSL certificate?

And what shall I use as base URL for Jira and confluence?

@Craig Castle-Mead for the "Path Pattern = /s/*", you said that you used "Cache Based on Selected Request Headers = Whitelist". Can you please share the whitelist? and also share what you are using as a base URL for Jira (is it your domain name of the ec2 custom domain name)

thank you vm

[M Amine]

Thank you for your reply @Craig Castle-Mead. 

Let me give you come more details about the infrastructure I'm managing:

• Cloudfront
• Origin: EC2 Instance in a public Subnet
  • NACL & SG (allowing only 80/443) + will add the lambda function (thanks ;) )
  • Nginx (reverse Proxy): forwarding 80(http)->443(https) + SSL + ... etc
  • Crowd
  • Jira
  • Confluence
  • ...
• RDS on a private Subnet

Do you think that I need to add the LB?

Besides, I'm still not able to deploy the CDN. I went through all the steps here but still getting 502 error page. I'm missing something that I cann't figure out !

Here is my configuration:

CNAME: I'm using 'xxx.domainname.com' for which I have created a CNAME record in my host (OVH).

I have a generated a SSL certificate that I have uploaded to ACM and used in cloudfront. 

While troubleshooting I have done the following:

• Used an SSL checker (https://www.sslchecker.com/sslchecker) which tells me that the certificate and the chain were find but not the root. BTW other online SSL checkers say that everything is ok. Don't know if the issue is coming from here. Amazon CM finds that the private, certificate and the chain are OK. 
• Used Openssl
  • openssl s_client –connect domainname:443 –servername domainname ==> is OK
  • openssl s_client –connect domainname:443 ==> is giving me SSL HANDSHAKE ERROR. But Cloudfront is not configured to allow non SNI clients. So I think it is ok
• Checked my domaine with https://www.ssllabs.com/ssltest/ for ciphers & protocols. Everything seem to be ok. I can give more details if necessary

Really cann't find what I'm missing !!

Thank you for your help

kind regards

[Craig Castle-Mead]

Hey Robert,

Not sure what could be happening with your ttf/woff's - maybe worth putting a rule just for *.woff and *.ttf at the top of your rules and forcing them through without any cache to check if they at least load, and if that works, you can change the behavior so they then should be cached - at least this way you know exactly which rule you're diagnosing (actually, it may be worth adding a custom header to each of your behaviors (rule = 1, rule = 2 etc etc) so that you can view the headers of each file and it will tell you which rule the file is using)

CCM

[Craig Castle-Mead]

Hey Mohammed,

To ensure only CloudFront traffic is getting through to your environment, you'll want to have the security group automatically update based on the IP ranges CloudFront uses. As these can (and do) change, AWS have an SNS topic you can subscribe, this can then trigger a Lambda function that updates the security group inbound rules (write up at https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/ )

PS. Even if you only have a single EC2 instance, I'd suggest putting a load balancer in front it and only allowing 80/443 to the load balancer from CloudFront and then only allowing 80/443 to the EC2 instance from the load balancer.

CCM

[M Amine]

Thank you vm @Craig Castle-Mead for the post. 

Tried to do the samething but I'm wondering how would I disable any traffic not getting from the cloudfront distribution? 

In my case I'm using an EC2 instance with a reverse-proxy sending traffic to Jira & Confluence. 

thanks

MKA

[Robert Beckman]

To answer my own question, whitelisting "Origin" headers seems to do most of the trick, but I'm having issues with woff and ttf files for some reason. They end up cancelling entirely.

[Craig Castle-Mead]

Hi Itesh,

Sorry for the belated reply - totally missed the notification.

We did not change anything in the Jira config at all - it was just a bunch of network inspection in browsers and looking for URL patterns that returned consistent content and mapping these to behaviors with varying levels of caching. The default behavior enforces no cache so just sends the entire request (all headers, cookies, query strings, methods) back to the origin as if the CDN didn't exist.

CCM

[Robert Beckman]

This is a very informative list, but I'm a little stuck as to what headers you are whitelisting in the caching settings.

[Itesh Shah]

Hi,

Thank you for the revert to my query.

How are you hosting your instances I meant wether they are "On-prem in a datacenter" or "Hosting in the cloud (AWS/Google/etc) sorry that I did not clarify what i meant. Thank you though for letting me know that you are doing it in AWS.

We are having an "On-prem" situation and trying to use cloud front.
Any special changes that needs to be done on Jira other than enabling the dark feature? and any special attention that I need to pay for CloudFront configs?

Regards,

Itesh