Keeping tickets closed when requesters say "Thank you"

[#1]

Like many organizations (Atlassian included), we prefer to reopen tickets when a requester replies after closure in case they need further assistance. This is a better experience for both the responder and the requester as the context remains in a single request and the interaction can resume where it left off.

The difficulty is determining whether a requester needs something additional or is simply acknowledging the closure. This isn't a new problem (see here, here, here, and here), but the solutions all seem to point toward string or pattern matching. This works sometimes, but there are enough cases where it doesn't to confuse both requesters and responders. 

For example, if a requester replies with "Thank you," followed by a question, your automation rule may inadvertently keep the request closed. You could look for text after "Thank you," and reopen the request, but it's common to have lengthy signature blocks or disclaimers appended to the bottom of messages, so this would reopen messages that don't require any follow-up. 

Additionally, people express gratitude or acknowledgement in many ways. "Thank you," is just one of them. In a short survey of requests within our organization, we observed, "Thanks," "Thank you," "Awesome," "tHANKS," "Much appreciated," "Appreciate it," "appreciate you," "OK, great," "Will do," "I'll let you know if it happens again," among many others (not to mention responses that don't even contain words: 👍).

There are tons of additional possibilities, none of which require a follow-up. Whenever one of these comments isn't properly matched, another round of email is sent out to the requester, the responder, and any request participants, informing them that the request has been reopened and then closed again. Trying to build regex or string-matching conditions for every one of these options is impossible. What we really needed was a tool that could grasp context and natural language like an LLM.

We're also an Azure customer, so we started experimenting with a simple prompt and pasting the comment body into Foundry using a low-cost hosted OpenAI model. The results were surprisingly good. With minimal context, the model was able to identify whether the request required follow-up or was simply an acknowledgement. 

After a little bit of tuning, this is the prompt we landed on:

You are a technical support assistant for an IT department. You need to review customer responses to determine if follow up action is required. If action is required - meaning the text contains a question, request, task, or a statement indicating a desire to reopen the ticket, you should respond to the prompt with "Action Required." If the text does not require any action and is simply an acknowledgement, respond with "No Action Needed." Here is the prompt: <comment body>

The prompt tightly controls the output to one of two responses from the model: either "Action Required" or "No Action Needed," which allows us to operate on the response in Automation.

Our next step was configuring our automation rule to forward the comment body to the Foundry model. We did so using an authenticated web request action to send the prompt along with {{comment.body.jsonEncode}}.

Initially, this was configured in a passive mode where we could review both the comment and the verdict it rendered via email without affecting the existing workflow. In the first 100, 98 had the verdicts we expected. One API call to foundry didn't receive a timely response, and one result was ambiguous whether the requester still needed assistance. In both cases, Automation would have reopened the request for the responder to review, which we felt was the preferred outcome. 

Messages that were simply acknowledgements were properly assessed with "No Action Needed."

 

Messages that required follow-up were properly flagged as "Action Required," even when combined with appreciation for what was completed already.

 

It even properly detected reactions 👍

 

The last step was to fully integrate this into the process with a conditional statement: 

If matches {{webResponse.body.choices.message.content}} equals "No Action needed" 

This ensures that we require positive confirmation that no action is required. Any other response (or no response) will force the request into the ELSE condition which reopens the ticket.

 

We implemented this over a year ago, and we're satisfied the solution. We continued to monitor verdicts for a couple of months after it was fully integrated, and it consistently rendered the appropriate verdict. 

Additional considerations:

1. Cost was a concern initially, so we monitored that closely as well. We're a mid-sized organization, and this only applies to tickets where a comment was added after closure, not every ticket in our queue. This averages to about 1000 tickets per month. The total cost for the first year of operation was a whopping $0.61 in Foundry. We considered this well-worth the time spent building, testing, and integrating it into our workflow. Cost would need to be more carefully evaluated for significantly larger volumes of data.
2. Privacy is also a consideration. We reviewed the Foundry privacy policy and felt comfortable sending the data back and forth knowing the data was exclusively for our use.
3. Security is also a concern. It's possible a malicious actor could include additional instructions for the model in the comment body. Since the automation rule is looking "No action required," manipulating the model through prompt injection would result in an unexpected response, and the ticket would reopen as a result. The runbook doesn't post the verdict to the ticket, so this removes the risk of data leakage. 
4. Be sure to tightly control your outputs in the prompt (or agent instructions) and define your preferred failure mode for unexpected outputs. 
5. There may be an avenue for Rovo to handle this natively within JSM now. I haven't explored it because the existing solution is inexpensive and effective.

[Italo Lobato]

I was thinking on using Rovo inside of the automation for this, did you try it?

[Ryan]

@Italo Lobato, we implemented this back in late 2024, so it was before Rovo was built out. I noted in the last consideration above that we haven't explored it because our existing solution is cheap and effective. If I were to start over today, I would definitely look at Rovo first since everything would remain within the platform. I'm not sure how open-ended the Rovo actions are in Automation (i.e., can I just give it an arbitrary prompt like the one we used and operate on the reply).

I thought to write this post because of a recent interaction with Atlassian support where I replied with a "Thank you," after the ticket was resolved, and I received a couple more emails indicating the ticket had been reopened and then closed again. We've been very satisfied with AI as a solution for this, and if Rovo isn't an option, the implementation was relatively simple.

[Brandon Viertel]

Morning @Ryan ,

Like @Italo Lobato said, you could use a Rovo agent for this. You could create a Rovo agent built for evaluating the response for if more work is required or they're just thanking you/liking your message. In the automation, you would then invoke the corresponding Rovo agent with some prompt like "Evaluate if more work is necessary". Of course, you would then be replying on Rovo to determine whether or not something is reopened and something could get missed or misinterpreted by it, but I think it's smart enough to evaluate on that.

Thanks!

Brandon