Rovo Agent – Hallucination Peak in Commit & Repository Identification

[Yudhees]

Rovo is showing a high level of hallucination when analyzing Jira-associated commits and repositories.

The behavior observed:

• Initially, Rovo reported only 2 commits for the ticket.
• After I indicated that there were 50+ commits, Rovo changed the result and identified 59 commits.
• The 59 commits appears to be correct.
• However, when asked to identify the repositories involved in those commits, Rovo returned 6 repositories.
• Only 2 of the 6 repositories were actually correct. The other 4 repositories were unrelated/incorrect.
• After further interaction, Rovo now reports the correct 59 commits across 2 repositories.

The major concern is that Rovo is not reliably deriving repository information from the actual commit data. Instead, it appears to be hallucinating repository names and presenting them as factual information.

This is especially problematic because the agent is intended to generate technical implementation documentation. Incorrect repository identification can result in completely inaccurate documentation about impacted systems and components.

Expected: Rovo should identify repositories strictly from the actual commits associated with the Jira ticket and should not infer or fabricate repositories when the underlying data is unavailable.

Current: Commit count eventually became correct (59), but repository identification showed significant hallucination, with 4 out of 6 initially reported repositories being incorrect.

[Arkadiusz Wroblewski]

Hello @Yudhees 

You can now check things Rovo pretty easily. For any wrong responses, use the Debug response (just put { } under the Agent response).

Be sure to grab the request ID, agent or subagent, Language Model ID, the tools used, and their inputs and outputs.

Then, see what info the retrieval/tool actually gave back. If the tool outputs 6 repositories already, the problem might be with the Teamwork Graph, connector, or retrieval layer.

But if the tool shows just the right 2 repositories and the final response says there are 6, that points even more strongly to a grounding or hallucination issue with the model.

Also try to test that inside new conversation because we have couple candidates which could cause this. 

[ROVO-287] ROVO resulting in incorrect information when processing large amount of data - Create and track feature requests for Atlassian products.

[ROVO-714] Agent new Model - Agent Hallucinates and Fails Jira Search - Create and track feature requests for Atlassian products.

[ROVO-840] Manual Reset for Atlassian Rovo Chat Memory and Context - Create and track feature requests for Atlassian products.

Best,

Arek🤠