Compressing Projects With Very Large Attachments

[SRBR SE]

Hey guys, We are having difficulties managing projects in jira that have very large attachments. Most of the time they are image and video attachments. Is there any way that we can compress this type of project, reducing the occupation of these files on the disks. It was already necessary to carry out an addition of disks due to the very rapid increase in utilization. I used project archiving but it seems to me if I'm not mistaken not to compress the attachments. Just archive the project. If anyone has any tips or guidance on how to work in this specific scenario I would be very grateful for the help.

Thank you very much

[Craig Castle-Mead · Accepted solution]

Hi @SRBR SE 

I’m not aware of any Jira native methods to compress/optimise attachment size on disc.

a few things to consider though:

1. limit the maximum attachment file size in Jira itself and implement a method for storing larger files (like video media) in a non-Jira based storage system that has some integration with Jira so they’re easy to reference in the Issues
2. depending on the OS/file system type, there may be a “transparent file comprsssion” solution you can implement. Jira will see the files as they were uploaded, but the OS/file system implements some logic as the files are written/read that reduces the actual space on disc and as much as possible (https://klarasystems.com/articles/openzfs1-understanding-transparent-compression is one random page I found that talks thorough a ZFS style solution). We’ve found on our environment (Jira has > 7TB of attachments and growing), that there are often a number of duplicated files uploaded, so even a file system that de-dupes the storage (save one actual copy and have pointers to the primary file from any of the duplicate paths) could save some space. You should be able to figure out how bad your duplicate problem is by getting the md5 hash of your files, then counting how many of those hashes are not-unique
3. You’re correct that archiving a project will not do anything with that projects attachments. Archiving makes the project read only and removes the issues from the index (which powers search, boards, etc)
4. I’m not sure what infrastructure you’re running this application on, but if you have the option of deploying storage that has a lower cost/TB (which will often mean slightly lower performance), then you may be able to move certain projects attachment directories (that could be your archived projects, or projects that need a large number of videos) to the “cheaper” storage and then sym-linking those folders back in to the main attachment directory. We have implemented this in production on AWS, Ubuntu 20.04 application servers and multiple EFS mounts. Our goal was to lower the recovery time objective (RTO) in our disaster recovery process as EFS can take a long time to restore. We now have our “main” attachment EFS that we keep as lean as possible, it gets restored first and once that’s restored we can start Jira and users can start working, but will only be able to see/add attachments for projects on that drive. The secondary EFS volume is then restored, and once that’s done, the attachments will be available for the rest of the projects. While we’ve set this up in production, YMMV, as always, test on a non-prod system first. We have a 4 node DC cluster and ensuring all the nodes have the correct symlinks setup is straight forward for us as we have Puppet Enterprise enforce/handle all the mounts/symlinks.

Points 2 and 4 are also not supported approaches by Atlassian as far as I’m aware, however the goal is that they should be invisible to the application itself. 

CCM

[SRBR SE]

Hi @Craig Castle-Mead ,

Thank you very much for the points addressed which were very enlightening for the necessary adjustments.

SRBR SE

[Florian Bonniec · Accepted solution]

Hi @SRBR SE 

Archiving issue only removed them from the index. Usually I recommand to not store large attachment and limit the size on the system settings. JIRA is a project managment tool, large files should be store somewhere else and a link to this attachment should be added to JIRA.

Regards

[SRBR SE]

Hi @Florian Bonniec ,

Thank you for the tip.

SRBR SE