Community
Q&A
Jira Service Management
Questions
Rovo Agent Inconsistent Results When Analyzing Large Jira Backlogs

Rovo Agent Inconsistent Results When Analyzing Large Jira Backlogs

We are currently evaluating Rovo for analyzing our Jira backlog and have encountered several issues that we would like clarification on.

Our backlog contains approximately 1,000 issues. The Rovo agent is able to retrieve all tickets successfully using JQL; however, we are observing the following behavior:

1. The agent retrieves all matching issues but appears to analyze only a subset of them.
2. When presenting the output, Rovo is unable to display all tickets result in the generated table it generate anly some of them and shows (...) at the end of table.
3. Running the same prompt multiple times against the same JQL query produces different results. The set of tickets included in the analysis and the final output varies from one execution to another, even when no changes have been made to the underlying Jira issues.

We would like to understand:

* Are there any limits on the number of Jira issues that Rovo can analyze in a single request?
* Does Rovo sample issues when the result set is large?
* Is there a maximum context window or processing limit that affects analysis of large backlogs?
* Is therow output limit expected behavior, and can it be configured or increased?
* Is it expected that identical prompts against the same dataset produce different outputs?
* Are there any recommended best practices for obtaining consistent and comprehensive analysis of large backlogs (1,000+ issues)?

Our expectation is that, when the same JQL query and prompt are used against an unchanged dataset, the analysis results should be consistent across executions. We would appreciate any explanation of the observed behavior and recommendations for ensuring complete and repeatable analysis.

Thank you for your assistance.

2 answers

0 votes

Hi @Fatima AALLA

You are seeing current Rovo platform limits. I hit the same thing when testing on a 2k issue backlog.

Quick answers to your questions:

1. Are there limits on how many Jira issues Rovo can analyze in one request
Yes. Rovo can retrieve a large set with JQL, but the analysis step runs on a smaller context window. Today the agent only sends a subset of the retrieved issues to the LLM for reasoning. The exact number is not documented publicly, but from testing it is usually around 100 to 200 issues per run.

2. Does Rovo sample issues when the result set is large
Yes. If your JQL returns 1,000 issues, Rovo will fetch them, then sample a portion for analysis. It does not process the full set end to end. That is why you see the ellipsis at the end of the table. The table output is also truncated to keep the response readable.

3. Is there a maximum context window or processing limit
Yes. There is both a context limit for the LLM and a time limit for the agent run. Large backlogs hit both. The agent will stop once it reaches the token or time budget, so only part of the backlog gets analyzed.

4. Is the row output limit expected and can it be increased
Expected, yes. The table UI in Rovo has a row cap to prevent huge outputs. You cannot configure or increase it today. If you need the full list, ask the agent to export to CSV or create a Confluence page. Even then, the underlying analysis may still be on a sample.

5. Is it expected that identical prompts produce different outputs
Yes, right now. Because Rovo samples the issue set, each run can pick a different sample. The LLM is also non-deterministic. So with 1,000 issues and no changes, you will get different tickets in the analysis each time. This is a known gap for audit or compliance use cases.

6. Best practices for consistent and comprehensive analysis of 1,000 plus issues
Use a two step approach:

Step 1: Reduce the dataset before analysis. Add more JQL filters like project, label, status, created date. Aim for under 200 issues per agent run. Run the agent multiple times on smaller slices, then merge results.

Step 2: For full backlog analysis, use a Forge Action or Automation rule. Have Forge pull all 1,000 issues with pagination, do the heavy counting or grouping in code, then pass a summary to Rovo. Let Rovo write the narrative on the summary, not the raw 1,000 issues.

Step 3: Ask for structured output. Tell the agent to group by component or priority first, then drill down. This reduces randomness.

Step 4: Export. If you need the raw list, say Export all matching issues to a CSV with key, summary, status and have Rovo call the Jira search API. The export uses pagination and will get all rows, even if the analysis was sampled.

Your expectation is valid, but the product is not there yet for full deterministic analysis of large sets. Atlassian has this on the Rovo roadmap. Search Rovo large dataset analysis in the Atlassian Developer Community and add your vote.

For now, treat Rovo as best for summarizing 100 to 200 issues at a time. For 1,000 issues, pre-filter or push the heavy work to Forge and let Rovo handle the write up.

Hope that helps.