How is the attachment storage path constructed in JIRA?

[Kristofer Hansson Aspman _Riada_]

Hi!

I am wondering about the attachment storage path for JIRA. On a server that I am running, the structure of the attachments directory (/var/atlassian/application-data/jira/data/attachments/) looks like this:

`-- JSDT
    `-- 10000
        `-- JSDT-2
            |-- 10100
            `-- thumbs
                `-- _thumb_10100.png

I am currently writing a groovy script in which I need to determine the path to an attachment. In answers to questions such as https://answers.atlassian.com/questions/297697, PathUtils.joinPaths() is used to construct the path. However, it does not seem to add the "10000"-directory seen above. My questions related to this are:

• Why do I have that directory? I.e. has there been a change to the directory structure since the answer to the above linked question was written? I am running JIRA Software 7.1.2 (with JSD) and can't really remember how it looked before.
• Where does the number 10000 come from? When is a "20000"-directory added? What is the logic behind it and its relation to the attachment IDs.
• Are there any methods that can be used to access this?

I bet I am missing something trivial, but what I see is a bit inconsistent with what I've found when looking around answers and documentation.

[crf · Accepted solution]

We have gone through several different directory structures for attachments over the years.  Although it may seem pointless on a small system, consider for a moment that there are JIRA instances out that that have tens of thousands of issues in a single project.

Operating systems and their filesystems impose certain limits on the number of files that can be contained in a single directory.  On ext3, for example, you may not have more than 65,535 entries in a single directory.  If we used just the issue key, then as your product grew:

`-- JSDT
    |-- JSDT-1
    |  ...
    `-- JSDT-65535
    !!! JSDT-65536 !!!

At some point you wouldn't be able to attach files to any new issues because the directory entry slots would be exhausted.  This has happened before:

https://jira.atlassian.com/browse/JRA-19873

https://jira.atlassian.com/browse/JRA-23758

We changed the layout in 7.0.0 (and JRA-19873 probably should have been marked as fixed as a result).

The "10000" is a grouping number based on the issue ID (meaning the database row's ID, not the issue number that goes after the project key).  Issues that get IDs from 10000 - 19999 go into the "10000" bucket.  Those from 20000 to 29999 go into the "20000" bucket.  This way there is a worst case of 10,000 issue directories in a single bucket, which is still within the limitations set by all of our supported operating systems.  Other points:

• The JSDT there is the original project key.  If you rename / rekey the project, the original one will still be used for this.
• The 10000 comes from quantization of the issue ID, not the attachment ID.  The attachment ID is not related to it.
• We consider the filesystem layout of attachments an implementation detail, not API.  It could change at any time, even in a minor release.
• If you absolutely need to do this, there are a few APIs you can use to get to it.  Most of these are deprecated because there is no longer any guarantee that attachments live in the filesystem at all (in Cloud, for instance, they don't), but AttachmentUtils.getAttachmentDirectory is probably closest to what you want.

[Kristofer Hansson Aspman _Riada_]

Thanks for the detailed explanation!

[MattS]

I believe that the upgrade to JIRA 7 does not change the existing directory structure, just affects new attachments

[Ash]

Yes, the upgrade doesn't affect previous file structure, new attachments will be created in new grouping folders.

[Emily Roger]

This information is false; the bucket has nothing to do with the issue ID. It is predicated on the issue number (issue num), which is found in the Jira issue database table. Other than deciphering the original issue key, there doesn't seem to be any other means to obtain this number through the rest services.

[Jakub Musialek]

Hi,

Kristopher did you manage to do the script, which you can share?

[Kristofer Hansson Aspman _Riada_]

Hi!

I haven't run the script since I originally posted my question a couple of years ago so I cannot guarantee that it still works in newer versions of Jira. I still had it saved though so I have pasted a stripped-down (untested) version of it.

You will have to add your own code for copying/modifying the attachment files, this merely fetches the path of each attachment and logs it.

Note that you will also need to define the issue variable if you're not running this from a context in which it is already defined (e.g. a ScriptRunner PF).

Also note that I have not tested the script below. Use it at your own risk and test in a test environment! ;)

import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.util.PathUtils

/* Get the attachment manager used to fetch attachments from Jira issues */
def attachmentManager = ComponentAccessor.getAttachmentManager()
def pathManager = ComponentAccessor.getAttachmentPathManager()
log.debug("Path to attachments: " + pathManager.attachmentPath)

/* Iterate over the Jira issue's attachments */
attachmentManager.getAttachments(issue).each {attachment ->

  /* The grouping ID is used by Jira to solve problems related to number of files in a directory */
  def groupingId = Math.round(Math.floor((double) attachment.id / 10000)) * 10000

  /* Path to the directory holding the JIRA issue's attachments */
  def dirPath = PathUtils.joinPaths(pathManager.attachmentPath,
                                    issue.projectObject.key,
                                    groupingId.toString(),
                                    issue.key)
  log.debug("Will look for attachments in " + dirPath)

  /* Path to the actual attachment */
  def filePath = PathUtils.joinPaths(dirPath, attachment.id.toString())
  log.debug("Path to Jira attachment: " + filePath)

  /* atFile will be copied to a temporary file as an intermediate step */
  def atFile = new File(filePath);
  if (atFile.exists()) {

    // Do something useful with the file here, e.g. copying it

  }

}

[Ramesh Kasavaraju]

instead of attachment.id use issuenum from jiraissue table or from MutableIssue use getNumber()

[Pascal Robert]

@Ramesh Kasavaraju , try that, but it seems the rouding have change in Jira 8.5 (or maybe in 8.0). 

For example, for issue 82190, 

 def groupingId = Math.round(Math.floor((double) issuer.number / 10000)) * 10000

will return 80000,  but in the file system, it's 90000. So you must use Math.ceil instead of Math.floor

[Eric Salonen]

Please however note well, that having

issue.projectObject.key

 in the path builder does not necessarily reference correctly the actual path of the attachment as if a project has been set with a new key, the folder still remains the original key, not the new key.

[James Guerin]

"The "10000" is a grouping number based on the issue ID (meaning the database row's ID, not the issue number that goes after the project key)."

This information is incorrect, the bucket is not based on the issue ID at all. It's based on the issue number (issuenum) which can be obtained by looking at the DB table jiraissue. There appears to be no way to get this number through the rest services, other than parsing the original issue key. However, as noted above, moving issues may create complications in this regard and so that shouldn't be considered a reliable source for obtaining this path.