STARTLING results from a Jira and Confluence Upgrade

Mike Rathwell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 13, 2018

Introduction

This post depicts the back end performance improvements found after the Jira and Confluence upgrade of Sept 8 2018. With this upgrade we:

  • Upgraded Jira 7.8.2 to Jira 7.11.2
  • Upgraded Confluence 6.8.2 to Confluence 6.10.2

Aside from upgrading to keep the applications current along with getting improvements and bug fixes, the expectation/hope was that we would resolve issues we were seeing with Jira. Jira was exhibiting a continuous growth in container memory utilization that, when it hit ~75%, would become sluggish or even unresponsive enough that it was unusable. Restarting Jira would always clear this. Worked but a bit of a PITA. After reviewing all the Atlassian release notes between the then current versions of Jira and Confluence and the target versions, there were several bug fixes that didn't speak exactly to what we were experiencing but suggested that they might be related to them.

The Environment

  • Application Implementation and Configuration
    • The Jira and Confluence implementations here are running containerized in Docker.
    • The images are designed to be rigorously stateless and location agnostic.
    • Built on a Debian 9 base image which has the latest Oracle JRE rather than relying on the JRE provided by the Atlassian "install" binaries.
    • No binary application installer used; applying the Atlassian tarballs to the base image.
    • Upgrades are extremely crisp. Essentially one entire version is removed and one totally new one put in place. No upgrade in place. The only thing that changes is the Jira/Confluence application version.
  • Operating Environment
    • "Docker" run environment is the AWS implementation using:
      • Elastic Container Registry (ECR)
      • Elastic Container Service (ECS),
      • EC2 Container host registered to ECS
        • Both Jira and Confluence are on a single m4.2xlarge EC2 container host
    • RDS databases using the PostgreSQL 9.5.x personality
      • Jira and Confluence are each on their own t2.medium RDS instance
  • Application sizes/licensing
    • Both Jira and Confluence are 500 user instances
    • Jira is an exceptionally busy instance with nearly all licenses consumed and is pounded all day, every day by a large percentage of that user base.

The Results

As noted in the introduction, my expectation/hope was that we'd at least solve the problems with the Jira instance. What I did not expect was just how big the improvement would be. After running for ~1 working week on the new versions to capture meaningful empirical evidence , this is what we're seeing on the back end...

In all images below, what we're looking at is a 2 week span from the AWS CloudWatch  console. The upgrade was started at 23:00 on Sept 8 and, with initially updating a bunch of plugins in both Jira and Confluence along with a couple of other maintenance items completed "officially" on Sept 9. This is where the delineation can be seen.

Overall Container Host Utilization

The before and after for both CPU utilization and network I/O is startling. A very clear and large drop. These are the graphs for the single m4.2xlarge EC2 instance running both Jira and Confluence.

Upgrade Results - EC2 CPU and Network.png

Database Utilization

As with the EC2 instance, we see both CPU and I/O having a clear drop at the Sept 09 delineation point. What is interesting is that the CPU drop for Confluence was greater than for Jira but Jira had a far more significant I/O drop

Upgrade Results - RDS utilization.png

Another interesting statistic is that the connections also changed rather a lot. In our environment, Jira is configured for a minimum of 20 connections and can span to 60. Conversely, Confluence is configured for a minimum of 0 and can span to 60. As such we see a clear drop in the number of connections consumed. I suspect, were Jira configured the same, we'd see the same with it.

Upgrade Results - DB Connections.png

Container Utilization

The discrete container memory and CPU utilization also have exhibited these same significant changes.

While Jira had only been up for about a week before the upgrade, we can see the gradual growth in memory utilization that would have eventually have led to needing to restart the instance. After the upgrade, it's been a nice, steady, flat utilization lower than where Jira usually started out with.

Confluence changes were a pleasantly unexpected change. While both CPU and memory utilization were flat, they both had very significant drops after the upgrade.

Upgrade Results - Container CPU and Memory.png

1 comment

Timothy
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 15, 2018

Would love to see the broken images.

Mike Rathwell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 18, 2018

Hey @Timothy,

That was actually the part that was most startling...

The images are identical other than the versions of Jira and Confluence in them. I have my image build put together such that I specify Jira/Jira Service Desk and Confluence versions as a variable which then goes out to pick up the associated tarball and plop it in on top of my base image.

The base image did not change at all; still the same Debian base and JDK that was under the previous versions. The container task definition in ECS is identical as well other than merely changing the task name (for visibility) and the image it picks up.

As such, I was rather (pleasantly) surprised to see the dramatic change.

That said, I will look through my dockerfile specs to see if there is anything in there that exposes something I don't want exposed and post them here. I don't think there is as I was rigorous in making them stateless and location agnostic. If nothing else, I have a great set of dockerfiles that builds rock solid production images.

Timothy
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 19, 2018

I meant to say that the images are broken for me. They seem to link to a Confluence page that requires authentication.

Mike Rathwell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 19, 2018

@Timothy see if you can see those now... my stoopid; copy/pasted this directly from my conf instance and i guess it put in links to the images rather than the images

Timothy
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 19, 2018

Nope. This is how it looks for me.

screenie.png

Mike Rathwell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
September 19, 2018

@Timothy How bout now? I did the strict "insert photos" thing this time...

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events