How to manage Jira index files in an AWS ECS deploy

Question

We're deploying the official Docker image of Jira (v8) into AWS ECS, but I'm having trouble finding a way to manage the index files - every time the ECS tasks restarts we lose our index files and have to manually reindex.

We use NFS volumes to store persistent data (attachments, etc), but the latency on this proved far too high to hold the indexes, so they just live in the docker container.

The trouble comes when ECS decides to rebuild our container (e.g because a Jira update is available, often just because it feels like it). The container is destroyed, along with the index files, and I have to manually reindex. Our instance is still quite small so it doesn't take long, but it's a manual step in an otherwise fully automated process.

Since our Jira container always runs on the same ECS host I've tried mounting the index directory (/var/atlassian/application-data/caches/indexesV1) in a shared docker volume so it persists between ECS tasks. This led to a permissions error, so I'm assuming Docker is creating the Jira home directory early on, before Jira starts, with the wrong permissions.

JIRA couldn't create the jira.home directory. Ensure JIRA has permission to create and write to the jira.home directory /var/atlassian/application-data/jira.

I'd rather not write our own Docker image, or need an external script to trigger a reindex. I feel like there's a more elegant solution to this that I'm missing.

Paul_ymrfxf · Answer

Solution:

Volumes:
      - Name: "caches"
        EFSVolumeConfiguration:
FileSystemId: "fs-xxxxxx"
          RootDirectory: "/caches"  # Not /indexesV1

[...]

MountPoints:
 - ContainerPath: "/var/atlassian/application-data/jira/caches"   # Not ...jira/caches/indexesV1
   SourceVolume: "caches"

The issue comes from the fact the caches folder, from /var/atlassian/application-data/jira/caches/indexesV1,does not exist on the container yet when running it.

Docker creates that 'caches' folder it with root permissions, preventing Jira's user (jira) to access it.

From the bind-mount documentation, https://docs.docker.com/storage/bind-mounts/, I cannot find a reference to a statement that Docker will create a folder on the container if it doesn't exist when mounting a volume.

Also, the host folders used as mounts must be created with the right permissions for Jira to be able to access them when mounted on the container:

JIRA_UID=2001
JIRA_GID=2001
mkdir caches
chown -R JIRA_GID:JIRA_UID caches
chmow -R 0700 caches

pskillen · Answer

Hi Andy, thanks for the suggestion but no luck, unfortunately. I still get the same error as above, using either a Docker volume or a bind mount.

I suspect this is actually a Docker issue rather than an ECS issue (or else it's something very specific about ECS' implementation of Docker).

I've included the script we use to start Jira below in case anything obvious stands out. We differ from the official documentation because we don't put all of /var/atlassian/application-data/jira into a single volume (because we need a mixture of persistent storage and low-latency storage).

I'll see if I can replicate this with a docker-compose file, which might help narrow down the problem.

ECSTaskDef:
Type: AWS::ECS::TaskDefinition
Properties:
Family: "jira"
    Volumes:
      - Name: "data"
        EFSVolumeConfiguration:
FileSystemId: "fs-xxxxxx"
          RootDirectory: "/data"
      - Name: "plugins"
        EFSVolumeConfiguration:
FileSystemId: "fs-xxxxxx"
          RootDirectory: "/plugins"
      - Name: "export"
        EFSVolumeConfiguration:
FileSystemId: "fs-xxxxxx"
          RootDirectory: "/export"
      - Name: "indexesV1"
        Host:
SourcePath: "/etc/dockerapp/jira/indexesV1"
      # Also tried this...
      # - Name: "indexesV1"
      #   DockerVolumeConfiguration:
      #     Scope: "shared"
      #     Autoprovision: true

ContainerDefinitions:
      - Name: "server"
        Image: "registry.hub.docker.com/atlassian/jira-software:8"

MountPoints:
          - ContainerPath: "/var/atlassian/application-data/jira/data"
            SourceVolume: "data"
          - ContainerPath: "/var/atlassian/application-data/jira/plugins"
            SourceVolume: "plugins"
          - ContainerPath: "/var/atlassian/application-data/jira/export"
            SourceVolume: "export"
          - ContainerPath: "/var/atlassian/application-data/jira/caches/indexesV1"
            SourceVolume: "indexesV1"

Andy Heinzer · Answer

Hi,

I understand that you're using the docker deployment for Jira upon an AWS ECS system, and that ECS appears to be destroying the container so often that it is forcing you to re-create the indexes in Jira manually anytime this happens.

While I'm not an AWS expert, I think that you could get past this problem by using a Bind mount instead of a shared volume.  From that doc:

Some common use cases for bind mounts are:

* To provide persistent data volumes for use with containers

It sounds like this is something you would want in this case.  Try this and let me know if this helps.

Andy