Using Automation for Jira to de-duplicate emails

[#1]

One of my clients had a problem that I suspect is quite common for many Jira administrators. They are using Jira Service Management and have set up an email inbox. When a customer sends an email to the inbox, they are also CC'ing a group of other people about the issue or incident.

Some of those on the CC list are replying to the original email with reply-all, causing  another email to be send to their JSM project. Since the reply doesn't include either the Issue Key or the email header that JSM uses to correlate emails to issues, JSM sees this email as a new issue and creates a ticket for it. This can cause a large number of duplicated tickets for the original email if participants are responding to a lengthy email trail that uses the original issue instead of the Jira-generated reply.

We came up with a workaround using Automation 4 Jira that you can see below;

(1) This rule is triggered when a new issue is created.

(2) It uses the following Smart Value to create a new variable called emailSubject. 

{{issue.summary.remove("[EXTERNAL]").remove("RE: ").trim()}}

We are removing the words "[EXTERNAL]" and "RE: " as those are added by the email system for replies and for certain emails that are generated by their mail system. This allows us to see 

(3) We check to see if there are any existing issues that match the email subject minus the [EXTERNAL] and RE: strings.  We restrict it to the last 12 hours so that we aren't accidentally de-duplicating new incidents that happen to use the same subject line.

If we find any matching issues, we transition the new issue to Completed and set the resolution to Duplicated. 

(4) Finally, we look for that original issue (there should be only one) and (5) add a comment to it with the body of the email. We may end up with very long comments, but that is ok for now.

This isn't a perfect solution:

• It creates a lot of Completed issues that will not be reviewed
• It runs the risk that legitimate new issues are de-duplicated and appended to the original issue
• It runs the risk of creating new issues if the string of emails crosses the 12 hour boundary
• If someone changes the subject line, we will get a new ticket

However, it should reduce the number of duplicates that need to be reviewed and manually closed as duplicates. 

I am interested in whether you have run into this problem and how you solved it. Your comments are welcome

[Bill Sheboy]

Hi @C_ Derek Fields 

Thanks for the write-up!  Have you noticed any timing issues between your steps 3 and 4, where the just completed issue gets picked up by the branch?  I wondered due the asynch processing of branches with more than one issue.  If so, might a re-fetch after the transition and prior to the branch slow things down enough to prevent this?

And regarding your 12 hour issue, what do you think about using a Lookup Issues on JQL with an ORDER BY Created to grab the first matching issue.  You could then know the oldest and decide how to act, perhaps adding links between the duplicates or to track down any accidentally closed new issues (e.g. re-open).

Best regards,

Bill

[Jack Brickey]

nice trick. i might try but take a more cautious approach initially, e.g. marking issues for inspection.

[mararn1618 _secretbakery.io_]

Hi @C_ Derek Fields, I think this is really cool.

I am a bit hesitant about Summary ~ "{{emailSubject}}", since it might catch completely separate incidents (e.g. "internet down", "my internet is down"). Do you experience problems like this? How well does this work for you?

I own a duplicate finder plugin and was thinking about making the duplicates- and similarity data available via JQL, so people can build other usecases with automation and filters. What do you think about that?

[C_ Derek Fields]

@Bill Sheboy - We haven't been running it long enough to see any race conditions. The Re-Fetch suggestion is a good one and I will incorporate it.

@mararn1618 _secretbakery.io_ As I indicated, the solution isn't perfect. The subject line of the emails would have to be virtually identical for the JQL to find what are essentially unrelated issues. It is the risk that we need to take to try to reduce the dozens of duplicates that we are seeing from people who are including the service desk inbox in their replies to the original email.

I greatly appreciate the thoughtful feedback.

[Jack Brickey]

I also wonder if Atlassian might be looking to extend the capabilities of the new “similar issues“ that we see with your service management issues. 

[Ricardo Fonseca]

Thank you so much! You are awesome! I was trying to do exactly that, on a very frustrating way.

[richard_korck]

@C_ Derek Fields thank you so much for this automation. It has really helped. If you have made any improvements to the automation since this post I would be extremely grateful if you could share the automation. Thanks again.