Remove Jira Issue Attachments by MD5 Hash Redux

[#1]

Update: This Article is specifically for Jira Server Edition, not the cloud offering.

In my previous post Remove Jira Issue Attachments by MD5 Hash (which you should read before this one as I show how to set up the Script Listener as some other shtuff.) I showed how to remove attachments from JIRA based on the MD5 hash of the attachment.

I was feeling pretty good after writing that post and having eaten my doughnut. So, I went to tell a couple of my colleagues about it. This was their reaction …

So, you expect me to …

1. know what an MD5 hash is?
2. know how to get the MD5 hash of a file?
3. know where to find this script to add the hash to?
4. not mess the whole thing up in the process?

Um … uhh … yes? Ok, so maybe my approach isn’t super easy except to the programmer type. And now that I think about it I don’t want to have to be the one to always fix these. So, back to the drawing board. Let’s get this right.

So, I need to make it easy for others than myself to help maintain. Maybe if I made a way for my colleagues to take an attachment from an issue ticket and simply drop to a centralized storage location that could be scanned by the script … yeah that could work. It involves no knowledge of MD5 hashes or scripting and should be easy for pretty much anyone to do.

Now if I only had a location where we could place these attachments. A place that JIRA is able to scan. A place that all my colleagues have easy access to. If only such a place actually existed … hmm … oh, wait!! I could just have them attach the files to another JIRA ticket that will be used as a control ticket of sorts. Any attachments attached to this ticket would be compared against by the script and if a match is found then the issue attachment is deleted. (insert Handel’s Messiah playing in my head here)

The great thing is that most of my script doesn’t really need to be changed. All I need to do is specify a control ticket key in the script and have the script build the list of hashes based on that ticket. Here is my ticket …

And here is the new script. I’ve cleaned it up a little from the last version and removed a call to a method that is currently set as deprecated. It still worked even with the call, but best to get rid of that call before Atlassian removes the method altogether. Simply replace “{Project Key}-{Issue Number}” on line 12 with the issue key that holds your attachments to remove. So, if for instance the issue is in the FOO project and the issue number is 789 then that line would look like this …

def controlIssue = “FOO-789”;

import com.atlassian.jira.component.ComponentAccessor;
import com.atlassian.jira.issue.AttachmentManager;
import com.atlassian.jira.issue.attachment.FileSystemAttachmentDirectoryAccessor;
import com.atlassian.jira.issue.Issue;
import com.atlassian.jira.issue.IssueManager;
import java.security.*;

/***********************************************************************************/
/* This is the ticket that has the attachments on it to compare MD5 hashes against */
/***********************************************************************************/

def controlIssue = "{Project Key}-{Issue Number}";

/***********************************************************************************/
/*                                                                                 */
/***********************************************************************************/

/************************************************************/
/* Don't edit below this unless you know what you are doing */
/************************************************************/

// Get the attachment hashes for our control issue to compare against
def attachmentHashes = getAttachmentHashesFromIssue(controlIssue);

// Obviously we don't want to run this on the control issue ... only on other issues.
if(event.issue.key != controlIssue) {
    deleteMatchingAttachments(attachmentHashes);
}

public void deleteMatchingAttachments(List<String> deleteHashes){
    def issue = event.issue;
    def attachmentManager = ComponentAccessor.getComponent(AttachmentManager);
    def attachments = issue.getAttachments();
    def attachmentFile = null;
    def bytes = null;
    def md = MessageDigest.getInstance("MD5");
    def digest = null;
    def hash = "";

    // Loop through each attachment on the issue
    for(a in attachments) {
        attachmentFile = getAttatchmentFile(issue, a.getId());
        bytes = getBytesFromFile(attachmentFile);
        digest = md.digest(bytes);
        hash = String.format("%032x", new BigInteger(1, digest));

        // Compare hash to the list of hashes we don't want
        for(h in deleteHashes) {
            if(hash == h) {
                attachmentManager.deleteAttachment(a);
                break;
            }
        }
    }
}

public List<String> getAttachmentHashesFromIssue(String controlIssueKey) {
    def deleteHashes = [];
    def attachmentManager = ComponentAccessor.getComponent(AttachmentManager);
    def issueManager = ComponentAccessor.getComponent(IssueManager);
    def issue = issueManager.getIssueObject(controlIssueKey);
    def controlIssueAttachments = attachmentManager.getAttachments(issue);
    def attachmentFile = null;
    def bytes = null;
    def md = MessageDigest.getInstance("MD5");
    def digest = null;
    def hash = "";

    // Get hashes for all the attachments in the control issue
    for(a in controlIssueAttachments) {
        attachmentFile = getAttatchmentFile(issue, a.getId());
        bytes = getBytesFromFile(attachmentFile);
        digest = md.digest(bytes);
        hash = String.format("%032x", new BigInteger(1, digest));

        deleteHashes.add(hash);
    }

    return deleteHashes;
}

public byte[] getBytesFromFile(File file) throws IOException {        
    def length = file.length();

    if (length > Integer.MAX_VALUE) {
        throw new IOException("File is too large!");
    }

    def bytes = new byte[(int)length];

    def offset = 0;
    def numRead = 0;

    def is = new FileInputStream(file);
    try {
        while (offset < bytes.length && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
            offset += numRead;
        }
    } finally {
        is.close();
    }

    if (offset < bytes.length) {
        throw new IOException("Could not completely read file " + file.getName());
    }

    return bytes;
}

public File getAttatchmentFile(Issue issue, Long attatchmentId){
    return ComponentAccessor.getComponent(FileSystemAttachmentDirectoryAccessor.class).getAttachmentDirectory(issue).listFiles().find({
        File it->
         it.getName().equals(attatchmentId.toString())
    });
}

And now my colleagues sing my praises (in my dreams) instead of cursing my name (which maybe still happens when I make hard to update workflows). Oh well, you live and learn.

[Davin Studer]

@Monique vdB Apparently when I submitted this I didn't categorize it as a JIRA article. Doesn't look like I can fix that. Are y'all able to categorize this?

[Monique vdB]

@Davin Studer no problem. You should be able to click the three dots at the bottom of the post and click Move.  (Dots are at the bottom for articles but at the top for everything else...) but I am happy to move it for you!

[Monique vdB]

@Davin Studer huh, that was super weird; it didn't give me Jira articles as an option (I'm guessing that's what you were seeing). I manually put in jira-articles and it worked. Weird.

[Davin Studer]

@Monique vdB Yep, that's what I was seeing. I just figured it was a permissions thing. Thanks.

[Monique vdB]

Well, I guess everyone who reads this article gets a bit of behind-the-scenes fun here down in the comments. 😆

[Ivan Tovbin]

We've had a similar issue with signature images flooding issue attachments. Managed to resolve it by checking actual file's height and width and remove that attachment if its height and width were smaller than predefined values. Not a 100% bulletproof solution but in practice it works perfectly 99% of the time. 

Here's the code I came up with:

import com.atlassian.jira.component.ComponentAccessor
import javax.imageio.ImageIO

int x = 0
for (int i = 1; i > 0 ; i++){
  if (issue.getKey().split("-")[1].toInteger() <= i*10000){
    x = x + i
    break
  }
}

def attMgr = ComponentAccessor.getAttachmentManager()
def attachments = attMgr.getAttachments(issue)

for (int i = 0; i < attachments.size(); i++){
  if (attachments[i].getMimetype().split("/")[0] == "image"){
    String filePath = "/opt/data/jira-data/data/attachments/${issue.getProjectObject().getKey()}/${x}0000/${issue.getKey()}/${attachments[i].getId()}".toString()
    def file = ImageIO.read(new File(filePath))
    if (file.getHeight() <= 90 && file.getWidth() <= 305){
      attMgr.deleteAttachment(attachments[i])
    }
  }
}

[Meg Holbrook]

This is a game-changer for my tiny help desk team! 

Our company uses 5-6 different social media icons below their signature lines, and it just eats. up. all. the. space. We've been looking for an easy solution like this for some time. 

I'm testing this now, but it looks promising!

[Meg Holbrook]

Quick question on this, since I am one of the non-programmer types you eluded to. 

What is the comparison looking for? Name of file, file size, something else?

I want to be sure that before I kick this off, only the images I specified are going to be removed no matter if the user has renamed them or if the attachment name in JIRA is different. 

[Ivan Tovbin]

@Meg Holbrook If you are asking about my solution, then first it checks if the attachment is an image by checking its mimetype. If it is, then it proceeds to construct a path to an actual file behind this attachment and then check that file's height and width against predetermined values (90 and 305 pixels in my case). If the file's height and width is smaller than these values, then that attachment is deleted.

Naturally, you might wanna modify the filepath depending on your Jira HOME directory location and also change the values to check a file's height and width against to better suit your needs.

[Davin Studer]

@Meg Holbrook My solution checks the MD5 hash of the email signature attachment(s) against the MD5 hashes of the attachments to the control ticket. An MD5 hash is a way to fingerprint a file ... any kind of file. The MD5 hash of each file is different so if you can compare the hashes and if they match then they are the same file. Now before anyone jumps on my case I know that this is not a 100% accurate statement, but for all intents and purposes it is. While it is TECHNICALLY possible to have two files with the same MD5 hash the likelihood of YOU ever having two files with the same hash is super unlikely. So, in regards to your question the file name does not affect this as the hash does not take file name into account, but rather the actual contents of the file. They could have two totally different names, but if they have the same hash they are the same file.

[Meg Holbrook]

@Davin Studer, a 99% solution is good enough for me in this scenario. We deal with internal customers only through this desk, so asking for a new/different image is acceptable.

One last question, will this script only run on new issues or all? 

I appreciate your assistance in breaking this down for me. 

[Davin Studer]

@Meg Holbrook Also, it should be noted that you can have two image files that are exactly the same size and to you look the same, but do indeed have different hashes. There could be slight differences in the pixels that are indistinguishable to the human eye, but when the computer calculates the hash they come up different. So, you may need to attach multiple versions of your social media images to the control ticket until you've weeded out all the variations that your team uses. Once you have them all you shouldn't get them on your tickets any more.

[Meg Holbrook]

@Davin Studer, that's great for us actually because we can attach new iterations of images and signature lines to the control ticket. 

[Davin Studer]

@Meg Holbrook In part one of this article I set up the ScriptRunner Script Listener to respond to "Issue Updated" event ... which is fired when the ticket receives an email reply. So, this will not cull out existing attachments. It will only fire when an issue is updated and only for that issue.

[Meg Holbrook]

@Davin Studer, that's great. I will visit the first part of the article. 

[Meg Holbrook]

Hey @Davin Studer,

Receiving an error when turning this on, can you point me to any configurations I missed? 

The only thing I updated in your code was the reference issue number. Apologies if I missed something glaringly simple! 

CorrelationId: 59829855-47a6-4cc1-89b8-5196b608c8fd
RUN Script identifier: dedb7c7a-20f4-45d7-ac8a-a4b716928876 com.adaptavist.sr.cloud.events.WebhookExecution (jira:issue_updated webhook fired for issue EHLP-20488) 
Script name: Remove Summit Social Media Images Took: 800ms Logs:
2018-03-12 19:52:01.362 ERROR - startup failed:
Script1.groovy: 4: unable to resolve class com.atlassian.jira.issue.Issue
 @ line 4, column 1.
   import com.atlassian.jira.issue.Issue;
   ^

Script1.groovy: 5: unable to resolve class com.atlassian.jira.issue.IssueManager
 @ line 5, column 1.
   import com.atlassian.jira.issue.IssueManager;
   ^

Script1.groovy: 1: unable to resolve class com.atlassian.jira.component.ComponentAccessor
 @ line 1, column 1.
   import com.atlassian.jira.component.ComponentAccessor;
   ^

Script1.groovy: 2: unable to resolve class com.atlassian.jira.issue.AttachmentManager
 @ line 2, column 1.
   import com.atlassian.jira.issue.AttachmentManager;
   ^

Script1.groovy: 3: unable to resolve class com.atlassian.jira.issue.attachment.FileSystemAttachmentDirectoryAccessor
 @ line 3, column 1.
   import com.atlassian.jira.issue.attachment.FileSystemAttachmentDirectoryAccessor
   ^

5 errors

2018-03-12 19:52:01.421 ERROR - Class: com.adaptavist.sr.cloud.events.WebhookExecution, Config: null

[Davin Studer]

I think I see what your issue is. By chance are you on Jira cloud?

[Meg Holbrook]

Yes. Which already tells me I should be sad because this won't work, right? Ha!

[Davin Studer]

@Meg Holbrook Yeah, unfortunately, this was written for the server version. I'll update the article to reflect that. It would take quite a lot to rewrite it for cloud ... not even sure if it would be possible for cloud, honestly.

[Meg Holbrook]

I've been looking for a solution for cloud going on two years now, this did provide me with a glimmer of hope though. 

I appreciate your work though. A wonderful script that I could still deploy on our server instance. 

[David]

@Ivan Tovbin where do you put that script to check for attachment image size and delete?

[Ivan Tovbin]

@David Using it in a scripted post function. You need an add-on for that (either Scriptrunner or JMWE).

[andrey mityukov]

hello, i've tried to test this with "Code runner" plugin what should be looks like same as you have offered to use. but nothing happend after I have added groovy listeneer with your code above

maybe you you have some ideas?

server version

[Davin Studer]

Well, specifically it was written for ScriptRunner, not Code Runner. But you could make it work with a minor tweak. Change line 26 from 

if(event.issue.key != controlIssue) {

to

if($event.issue.key != controlIssue) {

Then change line 31 from

    def issue = event.issue;

 to

    def issue = $event.issue;

I think that should do it. Everything else should be add-on agnostic.

[andrey mityukov]

Davin, a BIG thanks to you - thats worked! I have no idea that script running addons might work a little bit different. you idea and script is awesome!