How to configure JIRA to store attachments to an external service like Amazon S3?

[Sorin Sbarnea (Citrix)]

Maintaining attachments can be real resource hog as they tend to grow indefinetly and making the maintenance more and more complex.

Is there a way to outsource the attachments upload, storage and delivery to an external service la Amazon S3?

[Mikhail T]

There are ways to mount S3 buckets and make them appear as part of a filesystem -- on both Windows and modern Unix (using FUSE). So, in theory, one can mount a bucket and tell JIRA (and/or Confluence) to use it to store attachments. To access them one would use the web-server fronting the application (such as Apache) to either flat-out rewrite the app-generated HTML – modifying all attachment-links to point directly at S3 (with something like mod_substitute or mod_sed) – or to simply issue 301 (permanent) redirects in response to any requests for attachments (using RedirectMatch-directives provided by mod_alias).

Another alternative is to simply have a cron-job push new attachments over to S3 – outside of the application – and deleting the local file. Your redirect-rules would then have to be smarter – they'll need to check, whether the requested attachment is available locally and only issue a redirect, if it is no longer found on the host (in the assumption, that it was moved to the cloud already.) Apache's mod_rewrite can handle such logic with aplomb.

No, I have not done it – but it is possible in theory. It would've been much nicer, of course, if Altassian implemented this natively (S3 is not a truly POSIX filesystem, unfortunately – one may be unable to open, what was just written-out, for example).

But until our vendor cooperates, this is the best we can do...

[Tim Malone]

Not bad suggestions... there'll probably end up being a few edge cases to cater for but something like this could probably still be packaged up and used as a good second-best option! I particularly like the idea of cronning new attachments to S3 and then rewriting. If I get a spare day that's a good project...

[IT Admin]

Also interested in a plugin or update to integrate S3 into JIRA. Seems like common sense request with many benefits that falls in line with JIRA typical use (tickets with attachments, screenshots, error logs). Not sure why Nic doesn't agree.

[Nic Brough -Adaptavist-]

The problem hasn't really changed - typical Jira usage is small related attachments, screenshots and logs. If you manage them outside Jira, they lose their relevance. If they're large and numerous, well, that's not what Jira is for and it's not built for it, you should be using a different system.

[Tim Malone]

With all due respect Nic, I think you're misunderstanding the need. A blog might host it's pictures on S3 - to offload storage, speed up downloads, etc. - but that doesn't mean the pictures lose their relevance from the blog post. The pictures - or attachments - don't have to be big to make this useful.

It also then makes it a lot easier to practice immutability in servers. If I can throw away a server and reprovision it and not have to worry about managing the attachment storage, that's a huge maintenance win!

[Nic Brough -Adaptavist-]

I don't quite get what you think I have misunderstood.  I understand relevance of attachments to issues and that has nothing to do with the underlying storage.  I think you've read more into what I've said and not quite got what I was getting at (six years ago, but it's still right)

On the technical side - store the files where you want to.  Jira doesn't care what the underlying storage is, but does not support anything other than (what appears to be) a directory on a disk.  It's always been perfectly happy with immutability in servers.  My very first Jira server had its attachments on shared storage which moved a couple of times as the company shifted its resourcing.

On the procedural side, Jira assumes what you attach is for Jira's use.  It does not cope with you changing things without telling it.  I'm not sure that it would be fair to expect a system to do that.

[Tim Malone]

> It does not cope with you changing things without telling it. 

That's entirely fair and I think that's what this request is about. It would be nice to find a way to tell Jira about it :)

Many systems have (first or third-party) storage adaptors allowing S3 to be used for storage. The flow generally catches uploads and uses an appropriate AWS SDK to upload it to S3 rather than save it to disk (either proxying it via the server or uploading it directly client-side), and then stores the S3 location somewhere in metadata alongside wherever the attachment data would usually be shown.

When the attachment is downloaded by a user, this process is then hooked into again and the user is directed to the S3 URL (which can be pre-signed for security) to retrieve the file.

This both removes the load from the server plus lets other benefits kick in such as CDN etc. And then, as long as you have logs shipped off the server and the database is external, you can pretty much treat your install as ephemeral.

It doesn't have to be S3 either - plenty of options, S3 is just the most common probably because it was first-to-market.

Anyway, I've had a look around to try to see if anyone has written this already (either for Jira or Confluence) - there is this which goes a little of the way, but isn't quite the complete-replacement-of-attachment-storage idea that I think is being searched for in this thread (and it doesn't look at first glance to handle uploads either).

[geeqla]

Is there a manual or guideline somewhere about mounting of Amazon S3 as a storage for attachments? Also, would it be possle to mount S3 to a JIRA instance running on remote host outside Amazon Elasic Cloud? Would you recommend that or would it be better to keep everything hosted at Amazon? Thanks!

[Nic Brough -Adaptavist-]

All Jira needs for attachments is something that is presented to it like a file system it can read and write.

NFS, SAMBA shares, SAN storage, Dropbox, NAS, etc, all present areas that are directory like (whatever the underlying storage is) and work fine as attachment storage. Amazon S3 will work fine as long as it can be mounted and look like a file system to Jira.

[Sorin Sbarnea (Citrix)]

Probably you are missing an essential point: the whole point for using S3, Dropbox or similar solutions is to allow people to download the files directly form the provider, without requiring jira server to download and redirect the file.

Because they are using hashes for URLs, security is not so important as anyone with the URL should be able to download the file. And it is virtually impossible to guess links.

[Nic Brough -Adaptavist-]

Er, no, you asked about storing attachments on external services. I answered that - Jira just needs file system storage for attachments.

Managing attachments is a different question. What exactly are you looking for there? Why are you bypassing Jira when the whole point of attachments is that they belong to issues? If you want external document management, why not just disable attachments in Jira and do it with something designed for it?

[Sorin Sbarnea (Citrix)]

There are good performance reasons why anyone would want to just bypass jira for download attachments. Some people have to work with huge attachments, and in this case if you do not provide the link to the clould to the requester, jira will have to download the huge file from the cloud in order to pass it over to the client requesting it. This adds a considerably load and delays in the process.

Attachment management should be done by jira but jira should allow bypassing download links to 3rd party storage solution.

[Nic Brough -Adaptavist-]

I'm well aware that Jira isn't designed as a highly performant attachment server, but that's pretty much an irrelevance, there are other issues to worry about before you get to performance. As I already said, it's not designed as a document management system, so I'm trying to get at your actual requirements. Why are you trying to use Jira to do this instead of a document manger? It's aimed at small uploads like the odd picture and so-on.

[Sorin Sbarnea (Citrix)]

Thanks, Nic, I really apreciate your interest. In this case it is not about improperly using Jira for document management. In fact it is about the normal usage: to attach information related to issues, and sometimes this requires big logs, crashdumps and even memory dumps which can be huge. Most of the time these files are not changed or even accessed. Doing Jira backups can be really tricky if you store attachments locally so storing them in the clould should be much easier.

[Benny Chew]

We're also looking having this as well - currently we're just saving it to the EC2 host's EBS volume and it's not particularly ideal as it grows pretty fast!