Rovo has become a common way to ask Jira for a report, and so have Claude and ChatGPT connected through the Atlassian Rovo MCP server. Several Community threads this year describe the same surprise: the answer looked complete, it wasn't, and nothing said so.
This is the routine I use before an AI-produced Jira number leaves my screen. It takes about five minutes and needs nothing beyond Jira Cloud.
Full disclosure: I work on Advanced Export, a Marketplace app. I mention it once, in the last section. Everything before that works without it.
What people have run into
- A sample presented as the whole. A team asked Rovo to categorize roughly 600 service requests. The analysis kept stopping at around 40, and someone who tried to reproduce it still came up dozens short of the full count. (thread)
- A different answer each run. On a backlog of about 1,000 work items, the same prompt against unchanged data picked a different set of work items every time, and the table ended in an ellipsis. (thread)
- Two counts in one reply. A sprint agent gave five stories in one paragraph and six in the next. (thread)
- A capped tool. Through the MCP server, reading a work item returned only its oldest 20 worklogs. A monthly hours report came out about a fifth too low, with no error. (thread)
A related MCP search limit from the same thread was fixed in August 2026 (JRACLOUD-99221). Limits move, so a report that checked out last month still needs checking this month.
The causes are ordinary. The model reasons over what fits in its context, not over everything the search matched. Tools cap their responses. A number written in a reply is generated text unless a tool computed it. And as of October 2026, Rovo cannot read a work item's History tab (ROVO-329).
The five-minute check, worked through
Here is an illustrative example. The numbers are made up to show the steps.
You ask Rovo: How many open bugs does PLAT have, and how many of them are about performance? It replies with a table: 212 open bugs, 64 of them about performance.
- Get the denominator from Jira. Ask the agent which JQL it used, then run it yourself in issue search:
project = PLAT AND type = Bug AND statusCategory != Done
Search says 241. The answer is already 29 work items short, so it is a sample. - Add up the table yourself. Don't trust a TOTAL row the agent wrote. If its categories add up to 198 while it says 212, the table disagrees with itself as well as with Jira.
- Look at what the tool returned. Most MCP clients, Claude among them, let you expand each tool call and see the raw response. Count the keys that came back, and look for a total that is larger than the number of work items returned. Asking the agent "did you read everything?" is not the same check: its answer is generated too.
- Open three work items from the biggest group. Ask for the keys behind "performance", open three, and read them. A wrong one tells you how the categories were guessed.
- Ask again in a fresh chat. If 64 becomes 71, the agent is sampling. If it stays 64, that is encouraging, not proof: two runs can miss the same work items.
- Check as the person who will present it. The agent sees what the person asking can see. Issue security and project permissions can give two people different, correct totals for the same JQL.
Questions that need history
"How long did bugs sit in review?", "how many were reopened?" and "what was added after the sprint started?" all need the changelog. If the agent cannot read it, any figure it gives is inferred.
JQL covers some of these. This lists work items moved from Done back to In Progress in the last 90 days:
project = PLAT AND status CHANGED FROM "Done" TO "In Progress" AFTER -90d
It counts work items, not reopen events, and it matches status names, so add an `OR` clause for each other done status your workflows use.
For sprint figures, use the board's Sprint Report as the reference. Three things trip up most comparisons:
- carried-over work keeps the old sprint in its Sprint field,
- "completed" means the board's last column at sprint close,
- work added after the start is marked separately.
Compute first, then ask the AI to explain
When the check fails, don't fight the prompt. Separate the counting from the explaining:
- Get the numbers from something deterministic: a JQL count, a Jira report, or a complete export into a spreadsheet. For an export, check that the number of unique keys matches the search total.
- Paste the finished summary table into Rovo and ask for the narrative: what stands out, what changed, what to look into.
The model is good at the second job and unreliable at the first.
Free-text categories, like the 600 service requests above, split the same way. Ask the AI to propose categories and keywords from a sample of 50 to 100 requests. Apply those rules to every request in a spreadsheet, first match wins, with an "Other" bucket. The category counts plus Other must equal the JQL total. Then hand the counts back to the AI to describe.
Keep a receipt with any number you share: the JQL, when it ran, whose permissions it ran with, and the file or table behind it. "Here's the query" settles an argument faster than a chat transcript.
Where Advanced Export fits
Full disclosure again: I work on Advanced Export. Its Rovo agent does the compute-first step for you: it loads every matching work item, up to 20,000, with their change history, and answers with SQL you can read, edit and schedule as a recurrent report. Advanced Export on the Atlassian Marketplace
How do you check AI-produced Jira numbers before you share them? I'd like to hear what has caught you out.
I drafted this article with help from an AI assistant and checked the claims and links myself.
