Jira Containerization, AWS ECS, and Reindexing

Question

We're looking to redeploy Jira Data Center on AWS ECS using Atlassian's official docker images (https://hub.docker.com/r/atlassian/jira-software) , and as a trial we've got most of the application up and running, but we're having an issue in that hard reindexing will cause ECS to terminate the node that's attempting to reindex.

The typical behavior of Jira is to stop responding to web requests while it performs a hard reindex, and in non-containerized environments (and likely other non-ECS containerized environments) we can control what happens (or doesn't happen) when a node stops responding - i.e. in our typical EC2 Autoscaling environment, we can suspend replacing unhealthy nodes. Given the amount of control (or lack thereof) in ECS, I'm wondering if anyone has a solution or workaround for this.

Tried this on both 8.3.1 and 8.6.1 and had the same issue, if it matters.

Joshua Reddish · Answer

I have kind of hacked a solution to this, by creating a second load balancer/ASG for an admin node, and instead of using /status for the target group health check, i use /favicon.ico, as it returns a 200 even during indexing.

Out of curiosity, have you been able to confirm your instance is using all the processors available to it in the container? I am kind of hard stuck as my Jira instance refuses to recognize available processors to be greater than 1.

Robert Beckman · Answer

There is a (half) solution to this that I know of. The issue is in the ECS networking, and the load balancer configuration specifically. Using ECS with a load balancer, AWS will manage health checks with the load balancer and automatically replace containers who fail the health checks.

So the way around that is to start a container specifically for reindexing using an ECS service that doesn't use the load balancer. The problem here is, if that task has the same configuration as the containers that use the load balancer, it's expecting to operate through a proxy, which causes issues logging in. The way around that is to use the API to trigger the index.

It works, but there's gotta be a better way.