Challenges with getting Rovo to pull quantitative data consistently

Question

Hi there, This is likely a user error, but I built a Rovo agent to look across our Jira instance and report on the progress of items (child-level and epic-level) that were transitioned to 'Done', 'Resolved', or 'Released' status during a specific period of time stipulated by the end user. Every time I run the agent, it receive different output. I'll run it once, and receive 14 items back. I'll rerun and receive 62 items back. Any advice? Here is my exact prompt: Generate an Executive Progress Report for the period September 17 to September 24.

To ensure 100% coverage, please follow these logic rules:

* Search for every single Epic that was transitioned to a 'Done' or 'Resolved' status during this period, regardless of its Delivery Instance, Team, or Label.
* Search for every Story, Task, or Bug transitioned to 'Done' during this period. Group these by their parent Epic (even if the Epic itself is still 'In Progress').
* If a completed Story does not have a parent Epic, list it under a 'General Delivery' section.
* Do not apply any filters or specific Delivery Instances. I want a total view of all completed work.

Corina Chevalier · Answer

Thanks, everyone! Yes, I asked it why there was a difference and it acknowledged missing items and gave a rationale; however, when I tried running it again. Still the same issue. I will try your approach, Josh, and see what happens.

Have a great weekend!!!

Arkadiusz Wroblewski · Answer

Hello @Corina Chevalier

What you're experiencing stems from known limitations in how Atlassian Rovo agents process JQL search results and perform arithmetic.

Rovo 396 - When Rovo executes JQL queries returning more than 50 items, it can fail to process all paginated result sets consistently, leading to inaccurate total counts across runs.

ROVO-535 (Counting & Grouping Errors): Even when the underlying Jira search retrieves the correct issue keys, Rovo can miscount, drop items, or group them incorrectly in the final output.

ROVO-509 (Numeric Aggregation): Rovo frequently struggles with accurate aggregation and mathematical sums across custom fields.

Best,

Arek🤠

Josh · Answer

Hi Corina, I don't think this is user error. When an agent is asked to "search for every single Epic", it runs its own searches, and how far it pages through the results can vary from one run to the next. It's still a language model, not a query engine, so 14 one time and 62 the next is sadly typical.

What has worked for me is to let JQL decide which items are in scope and only let Rovo do the writing:

* Epics: issuetype = Epic AND (status CHANGED TO Done DURING ("2026-09-17", "2026-09-25") OR status CHANGED TO Resolved DURING ("2026-09-17", "2026-09-25"))
* Children: issuetype in (Story, Task, Bug) AND (status CHANGED TO Done DURING ("2026-09-17", "2026-09-25") OR status CHANGED TO Released DURING ("2026-09-17", "2026-09-25"))

(A bare date means midnight at the start of that day, so use the 25th if you want the 24th included.)

* Run both in search first and note the counts. That gives you the number every report should hit.
* Save them as filters and put the exact JQL in the agent instructions: "Use only the results of this JQL. State the total count returned before summarizing, and list every key." If the key list comes back short, you'll see it straight away.
* For a fully repeatable weekly run, do it in Automation instead. Use a Scheduled trigger, then Lookup work items with the JQL, then pass {{lookupIssues.size}} and the keys/summaries into the Use Rovo agent action for the narrative. The count then comes from Jira, not the model. Lookup returns up to 100 items, so split by team or week if you go over that.

If this is going to be a standing weekly or monthly exec report, Aneto AI Executive Reports is built around that same idea. You set the scope with a JQL filter, pick the week or month, and can remove items before the AI writes the report. The item set comes from the JQL, not from the model's own search.

Disclosure: I work at Aneto Software, which makes this app.