Rovo Agent Inconsistent Results When Analyzing Large Jira Backlogs

[Fatima AALLA]

We are currently evaluating Rovo for analyzing our Jira backlog and have encountered several issues that we would like clarification on.

Our backlog contains approximately 1,000 issues. The Rovo agent is able to retrieve all tickets successfully using JQL; however, we are observing the following behavior:

1. The agent retrieves all matching issues but appears to analyze only a subset of them.
2. When presenting the output, Rovo is unable to display all tickets result in the generated table it generate anly some of them and shows (...) at the end of table.
3. Running the same prompt multiple times against the same JQL query produces different results. The set of tickets included in the analysis and the final output varies from one execution to another, even when no changes have been made to the underlying Jira issues.

We would like to understand:

* Are there any limits on the number of Jira issues that Rovo can analyze in a single request?
* Does Rovo sample issues when the result set is large?
* Is there a maximum context window or processing limit that affects analysis of large backlogs?
* Is therow output limit expected behavior, and can it be configured or increased?
* Is it expected that identical prompts against the same dataset produce different outputs?
* Are there any recommended best practices for obtaining consistent and comprehensive analysis of large backlogs (1,000+ issues)?

Our expectation is that, when the same JQL query and prompt are used against an unchanged dataset, the analysis results should be consistent across executions. We would appreciate any explanation of the observed behavior and recommendations for ensuring complete and repeatable analysis.

Thank you for your assistance.

[HEMANT SAINI]

Hi @Fatima AALLA 

You are seeing current Rovo platform limits. I hit the same thing when testing on a 2k issue backlog.

Quick answers to your questions:

1. Are there limits on how many Jira issues Rovo can analyze in one request
Yes. Rovo can retrieve a large set with JQL, but the analysis step runs on a smaller context window. Today the agent only sends a subset of the retrieved issues to the LLM for reasoning. The exact number is not documented publicly, but from testing it is usually around 100 to 200 issues per run.

2. Does Rovo sample issues when the result set is large
Yes. If your JQL returns 1,000 issues, Rovo will fetch them, then sample a portion for analysis. It does not process the full set end to end. That is why you see the ellipsis at the end of the table. The table output is also truncated to keep the response readable.

3. Is there a maximum context window or processing limit
Yes. There is both a context limit for the LLM and a time limit for the agent run. Large backlogs hit both. The agent will stop once it reaches the token or time budget, so only part of the backlog gets analyzed.

4. Is the row output limit expected and can it be increased
Expected, yes. The table UI in Rovo has a row cap to prevent huge outputs. You cannot configure or increase it today. If you need the full list, ask the agent to export to CSV or create a Confluence page. Even then, the underlying analysis may still be on a sample.

5. Is it expected that identical prompts produce different outputs
Yes, right now. Because Rovo samples the issue set, each run can pick a different sample. The LLM is also non-deterministic. So with 1,000 issues and no changes, you will get different tickets in the analysis each time. This is a known gap for audit or compliance use cases.

6. Best practices for consistent and comprehensive analysis of 1,000 plus issues
Use a two step approach:

Step 1: Reduce the dataset before analysis. Add more JQL filters like project, label, status, created date. Aim for under 200 issues per agent run. Run the agent multiple times on smaller slices, then merge results.

Step 2: For full backlog analysis, use a Forge Action or Automation rule. Have Forge pull all 1,000 issues with pagination, do the heavy counting or grouping in code, then pass a summary to Rovo. Let Rovo write the narrative on the summary, not the raw 1,000 issues.

Step 3: Ask for structured output. Tell the agent to group by component or priority first, then drill down. This reduces randomness.

Step 4: Export. If you need the raw list, say Export all matching issues to a CSV with key, summary, status and have Rovo call the Jira search API. The export uses pagination and will get all rows, even if the analysis was sampled.

Your expectation is valid, but the product is not there yet for full deterministic analysis of large sets. Atlassian has this on the Rovo roadmap. Search Rovo large dataset analysis in the Atlassian Developer Community and add your vote.

For now, treat Rovo as best for summarizing 100 to 200 issues at a time. For 1,000 issues, pre-filter or push the heavy work to Forge and let Rovo handle the write up.

Hope that helps. 

[Fatima AALLA]

hello @HEMANT SAINI ,

tahnk you for the provided answer, it's very helpful.

Yes I'm new to forge didn't work on it before, could you provide detailed steps or helpful links?

Thank you in advance for your support.

[HEMANT SAINI]

Hi @Fatima AALLA 

No problem at all. Since you’re new to Forge, here’s the full step-by-step to analyze 1,000 plus Jira issues with Rovo plus Forge. This bypasses the sampling and row limits.

Goal
Use a Forge Action to pull all issues with pagination. Do the counting or grouping in code. Send only the summary to Rovo for the write-up. This way Rovo doesn’t hit its 100 to 200 issue limit.

Step 1: Set up your Forge app

Install Node.js 18 or higher on your machine
Open your terminal and run the command npm install -g @forge/cli
Log in by running forge login and approve in the browser with your Atlassian account
Create app by running forge create, choose the Rovo agent action template, name it jira-bulk-analyzer
Step 2: Update manifest dot yml
Open the file named manifest dot yml in your app folder. Delete everything in it and paste the text below. This gives the app permission to read Jira issues.

Start of manifest dot yml content:

modules:
rovo action:
- key: analyze-large-jql
name: Analyze Large JQL Result
description: Pulls all issues for a JQL query and returns summary stats
function: resolver
inputs:
jql:
title: JQL Query
type: string
description: Example project equals ABC and status not equals Done
actionVerb: POST
function:
- key: resolver
handler: index dot handler

permissions:
scopes:
- read:jira-work
- search:jira
app:
id: ari:cloud:ecosystem::app/your-app-id

End of manifest dot yml content

Step 3: Add the code to src slash index dot js

In your terminal, run: npm install @forge/api @forge/resolver
Open the file src slash index dot js
Delete everything in it and paste the text below. This code handles pagination and returns a summary so Rovo doesn’t choke on 1,000 issues.
Start of index dot js content:

import Resolver from '@forge/resolver';
import api, { route } from '@forge/api';

const resolver = new Resolver();

resolver.define('analyze-large-jql', async ({ payload }) => {
const { jql } = payload;
let allIssues = [];
let startAt = 0;
const maxResults = 100;
let total = 0;

// 1. Paginate through all issues
do {
const res = await api.asUser().requestJira(
route/rest/api/3/search?jql=${jql}&startAt=${startAt}&maxResults=${maxResults}&fields=key,status,priority,components,created
);
const data = await res.json();
allIssues = allIssues.concat(data.issues);
total = data.total;
startAt = startAt + maxResults;
} while (startAt < total);

// 2. Do the heavy analysis in code, not in the LLM
const byStatus = {};
const byPriority = {};
const byComponent = {};
const over365Days = [];

const oneYearAgo = new Date();
oneYearAgo.setDate(oneYearAgo.getDate() - 365);

allIssues.forEach(issue => {
// Count by status
const status = issue.fields.status.name;
if (byStatus) {
byStatus = byStatus + 1;
} else {
byStatus = 1;
}[status]

// Count by priority
const priority = issue.fields.priority? issue.fields.priority.name : 'None';
if (byPriority) {
byPriority = byPriority + 1;
} else {
byPriority = 1;
}[priority]

// Count by component
if (issue.fields.components && issue.fields.components.length > 0) {
issue.fields.components.forEach(c => {
if (byComponent[c.name]) {
byComponent[c.name] = byComponent[c.name] + 1;
} else {
byComponent[c.name] = 1;
}
});
} else {
if (byComponent['No Component']) {
byComponent['No Component'] = byComponent['No Component'] + 1;
} else {
byComponent['No Component'] = 1;
}
}

// Flag old issues
const created = new Date(issue.fields.created);
if (created < oneYearAgo) {
over365Days.push(issue.key);
}
});

return {
totalIssues: total,
statusBreakdown: byStatus,
priorityBreakdown: byPriority,
componentBreakdown: byComponent,
issuesOver365Days: over365Days.length,
sampleOldIssues: over365Days.slice(0, 10)
};
});

export const handler = resolver.getDefinitions();

End of index dot js content

Step 4: Deploy and install

Deploy by running: forge deploy
Install by running: forge install, pick your Jira site, approve permissions
No Org Admin needed for Jira scopes. Any site admin can install.
Step 5: Use it in Rovo

Go to Rovo Studio, open your Agent, go to Actions, click Add action
Select Analyze Large JQL Result from your app
Now prompt your agent like this: Use analyze-large-jql with jql equals project = ABC and created less than -30d. Then write an executive summary of the backlog health using the returned stats. Call out risks from issuesOver365Days.
Why this works

Forge pulls all 1,000 plus issues using pagination. No 50 or 200 limit.
The code counts and groups everything. Rovo only gets a small JSON summary.
The LLM stays deterministic because the numbers are pre-calculated. No more random samples each run.
You can still ask Rovo to create a Confluence page or Jira tickets from the summary.
Helpful topics for learning Forge
Since Community strips links, go to developers dot atlassian dot com and search these:

Forge Getting Started
Forge Rovo Actions tutorial
Forge Jira REST API example
Atlassian Developer Community Forge topic
Biggest gotcha
If your JQL has special characters like quotes, pass the JQL from Rovo exactly as a string. The code above handles it, but test with a simple JQL first like project equals ABC.

The first deploy is the hardest. Once it’s working you can expand the code to calculate any metric you need.

Regards,
Hemant

[Arkadiusz Wroblewski]

Hello @Fatima AALLA 

Rovo / AI output is not guaranteed to be deterministic, so results can vary between runs.

Have you tried instructing Rovo to paginate the output or process the results in smaller batches?

This is quite similar to a topic I answered before.

Solved: ROVO provides inconsistent results

Use Rovo to search for work items | Jira Cloud | Atlassian Support

This was also already discussed and answered in depth in an older question, so it may be worth checking that thread.

Solved: Jira Automation + Rovo Agent: JQL Limited to 50 Is...

Best,

Arkadiusz🤠