nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How to get rovo to respond with JSON consistently?

Jeremy Wood

I am trying to build out an automation that uses a rovo agent to score a ticket based on set parameters, and then updates the custom field with the score as well as adds a comment with the reasoning that the agent used to reach that score.

I currently have my instructions set to "respond with a valid JSON object only". And then in my automation i use the smart values {{agentResponse.asObject.score}} and {{agentResponse.asObject.reasoning}} in my automation.

This works great the majority of the time but every now and then the rovo agent wont give a good json response. It will leave the brackets off the end of the json, or give a response along the lines of ```json { "score": 1, "reasoning": "….." } ```

I have tried updating the instructions over and over and this still happens randomly despite what i tell it.

Is there a better way to extract out the responses from the rovo agent that doesn't depend on rovo following its output instructions? Or a way to get it to more consistently provide a JSON output?

I know Atlassian is going to be adding the ability for Rovo to take actions directly on tickets in the future and that this is in early access right now. But i need a solution until then.

Accepted answers

All comments

Jeremy Wood

I just found that agent actions is actually starting to roll out to users. So perhaps i can wait for this solution.

Would still love to hear others feedback on how others were handling this problem though.

🚀 Now rolling out: Agent actions in Automation & Agent Accounts - Atlassian

Josh

Hi @Jeremy Wood , agent actions will be the clean fix once they reach you. Until then, the trick is to stop trusting the raw response and make the rule tolerant of the two failure modes you're seeing.

1. Strip the code fence before parsing. The usual failure is the model wrapping valid JSON in ```json … ```. Clean it up in a Create variable action (e.g. cleanResponse):
{{agentResponse.replace("```json","").replace("```","").trim()}}
Then read {{cleanResponse.asObject.score}} instead of parsing the raw response.

2. Validate before writing. Add an If/else block:

If: Advanced compare, {{cleanResponse.asObject.score}} matches regular expression \d+
Then: edit the field and add the comment.
Else: add a label like rovo-score-failed (or re-invoke the agent once). That way a bad response never writes garbage into the field, and you can sweep the failures with JQL.

3. Or skip JSON for the number entirely. Ask the agent to start its answer with a fixed line such as SCORE: 3, followed by its reasoning. Then pull the number out with {{agentResponse.match("SCORE:\s*(\d+)")}} and use the whole response as the comment. A regex on one fixed line is far more forgiving than a JSON parser when the model drifts.

In the agent instructions, a short literal example of the exact output usually helps more than repeating "JSON only".

Jeremy Wood

Hey @Josh. Thank you for the feedback. I tried getting these options to work today with no luck.

Both option 1 and 3 that you provided, the smart values just return blank.

Option 2 of adding a validator does work, and helps with tracking down the tickets that have failed. But doesn't really solve the problem of them failing to begin with.

Jeremy Wood

Update: Following Option 2 that josh proposed, but using it to re-invoke the agent again seems to have cut down on the number of failures.

I was shown agent response and how to use an agent within automation at Team26 back in May. So i feel like this was around long enough that i cant be the only person that had this issue with it. I wonder if i am doing something wrong still or if agent actions were born out of a need to address how unreliable this has been.

Arkadiusz Wroblewski

Hi @Jeremy Wood

Atlassian actually has a support article addressing inconsistent JSON responses from Rovo agents:

https://support.atlassian.com/rovo/kb/handling-inconsistent-json-responses-in-rovo-agent-prompts/

It explains how the main agent can modify valid JSON returned by a subagent.

also check the Debug response to identify where the formatting breaks and try {{agentResponse.asString}} instead of {{agentResponse}} to get plain text.

Neither guarantees valid JSON, so keep your validation and retry logic

I hope a lot of these problems will be solved in common time as Rovo get Attention last couple of months.

Best,

Arek🤠🧙‍♂️

Jeremy Wood

Thanks Arkadiusz Wroblewski. That agent specifically talks about sub-agents. I am just using a single agent so it wouldn't apply.

Using .asString may help address the problems i was facing with Josh's solution though. Good idea.

Jeremy Wood

I was also able to track down some older workshop handounts and videos on the topic.

(Handout) Let's Build a Rovo Agent: Jira Triage Automation

[Workshop] Let's build a Rovo Agent: JSM Triage

I noticed that in both of these the way they wrote their instructions were different from how Rovo tried to write up the instructions for this agent.

The workshop instructions show giving more details and specifics such as "JSON object with three keys:" and then list out what each specific key should be.

Where the rovo created instructions are much simpler and basically say return a JSON object with the below structure.

Both work at face value, but i wonder if the more detailed instructions will yield more consistent results.

I have updated my instructions this morning to follow more of the workshop models to see if they get better results.

Workshop instructions:

Rovo Generated instructions:

Arkadiusz Wroblewski

Then Rovo hallucinating because im not sure if <integer between 1 and 10> is valid JSON

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

rovo

automations

automation-smart-values

rovo-agent