Confluence running and AWS disconnected after Jira directory sync

Question

Hi,

I've setup the Confluence server on an aws t2.small and started migrating user data from a similar jira server on t2.small. However, now the confluence process is taking so much cpu that neither the ssh is working nor I can ssh into a different session. Please point me to a solution.

Thanks in advance.

Steffen Opel [Utoolity] · Answer

As Craig already pointed out (+1), you have most likely overstretched the performance profile of that T2 Instance, in particular due to running a long-lived process with constant CPU load. While T2 Instances have an excellent price/performance value for spiky workloads, their documented burst behavior is not suited for such long-running tasks, which seems to be what you are experiencing:

T2 instances are designed to provide a baseline level of CPU performance with the ability to burst to a higher level when required by your workload. T2 instances are well suited for a wide range of general-purpose applications like microservices, low-latency interactive applications, small and medium databases, virtual desktops, development, build, and stage environments, code repositories, and product prototypes. [emphasis mine]

If you want to learn more about how this works, the underlying CPU Credits and Baseline Performance model is well documented:

Traditional Amazon EC2 instance types provide fixed performance, while T2 instances provide a baseline level of CPU performance with the ability to burst above that baseline level. The baseline performance and ability to burst are governed by CPU credits. A CPU credit provides the performance of a full CPU core for one minute. [emphasis mine]

So once you have drained your CPU credits and depending on your specific workload, the instance will definitely feel considerably slower from a user perspective!

As a compromise to switching to a non-burstable instance type family (all others), you might want to look into the special T2 Unlimited feature, which can burst above the baseline for as long as required:

This [...] ensures that your instances are never held to the baseline performance. ​The basic T2 hourly instance price automatically covers all CPU usage spikes if the average CPU utilization of a T2 Unlimited instance over a rolling 24-hour period is at or below the baseline. [...] If the average CPU utilization exceeds the baseline over a 24-hour period, there is a flat additional rate per vCPU-hour. For more information, see the T2 Unlimited Pricing section in Amazon EC2 On-Demand Pricing.

That being said, given the migration task might still be a one time or at least rare load scenario, I'd rather follow a variation of Craig's advise and temporarily switch to bigger (possibly really big) instance types to get the migration over with really quickly - as long as you can accept the brief downtime for the switch, scaling vertically up and down again like so nicely demonstrates one of the main benefits of infrastructure as a service :)

Craig Castle-Mead_20fp9z · Answer

Hi Matthias,

A likely easy fix is to stop the node (when you can handle an outage) and increase the size of the instance. If this is a production instance or has growing traffic you may want to change from the t2 to m4 or m5 instance type family as you’ll get more reliable performance.

CCM