What I learned building a multi-agent Rovo reporting system and how I stopped it from hallucinating

[Todd]

I've spent the last few months building a Rovo agent that produces portfolio status reports across a large Jira instance — status rollups, kanban snapshots, sprint summaries, and release notes. Sharing what I built and the lessons in case they help others going down this path.

What I built

Rather than one big agent, I landed on an orchestrator + sub-agents pattern: a thin router agent that does nothing but hand off to focused sub-agents, one per report type. Keeping each sub-agent small and single-purpose made them far easier to keep accurate — and it sidestepped the config-size limits you hit when instruction bodies get too big.

Lessons learned

The biggest challenge wasn't wiring things up — it was trusting the numbers. Early versions confidently reported totals that were subtly wrong. What fixed it:

• Read counts off the query's total field. Never let the agent tally an in-memory list — pagination caps mean that list is usually incomplete. Paginate every query to completion.
• Group by status category (Done / In Progress / To Do), not by literal status name. Custom workflow statuses vary between projects; categories are stable.
• Define ambiguous words explicitly. Spell out exactly what "open," "remaining," or "complete" mean, or the model will guess. Mine once labeled the full descendant count as "open" when most were already done.
• Any breakdown must come from a query, not a guess. For by-type or by-assignee splits, run the actual count query. I caught the agent inventing a tidy two-type split that just happened to sum to the total.
• Add a reconcile step before output. Bake in checks the agent must pass (sub-buckets sum to the total, subsets stay within their parent). If a check fails, it re-queries instead of shipping.
• Put concrete failure examples in the instructions. Showing the exact past mistake works better than abstract rules.

Keep the orchestrator thin, keep sub-agents lean, and make the agent show its work so bad numbers are easy to spot.

Curious whether others have found similar accuracy patterns with Rovo — what's worked for you?

[MeghnaP_LogicLemur Labs]

Really interesting point about making the agent “show its work” @Todd . I think there’s another layer that becomes important once these multi-agent systems move into production: measuring the work the agents themselves are doing.

For example:

• How many Jira issues did an agent actually work on?

• How long did it spend?

• How often did a human have to take over?

• Did the output get accepted or require rework?

• Which agents are actually delivering useful engineering capacity?

Accuracy of the report is one side of the equation. Agent performance and contribution become the other.

I’m seeing this become an interesting gap as teams move from experimenting with Rovo to treating AI agents more like part of the engineering workforce.

Curious if others are starting to track these metrics alongside Rovo accuracy.

[Todd]

Great point — I'd love to see more analytics on that @MeghnaP_LogicLemur Labs . I haven't come across a tool yet that exposes all of it in one place.

We lean on LinearB to fill some of the gaps (PRs with AI vs. without, rework with AI vs. without, etc.), which covers the delivery side reasonably well. But the gap that remains for me is analyzing the prompts themselves — things like how often a prompt satisfies the user on the first try vs. requiring follow-up prompts to get there. That "did the agent actually land it" signal feels like the missing piece, and it's exactly where the human-takeover and rework questions you raised show up.

[MeghnaP_LogicLemur Labs]

Exactly. That “did the agent actually land it?” signal is the interesting part.

We actually built AgentWorkLog for Jira around this gap. It gives teams a way to see the work AI agents are actually performing in Jira, rather than only looking at the agent response itself.

For example, you can start looking at things like:

• which Jira issues an agent worked on

• what work was completed

• agent activity over time

• where human intervention was needed

• how much work is actually being handled by agents

So I see it as a layer between Rovo observability and engineering delivery metrics.

LinearB is useful for measuring the delivery side. AgentWorkLog is more focused on answering: “What did our AI agents actually do?”

Would be interested in your take on whether that fills the gap you're describing.