Rovo is showing a high level of hallucination when analyzing Jira-associated commits and repositories.
The behavior observed:
- Initially, Rovo reported only 2 commits for the ticket.
- After I indicated that there were 50+ commits, Rovo changed the result and identified 59 commits.
- The 59 commits appears to be correct.
- However, when asked to identify the repositories involved in those commits, Rovo returned 6 repositories.
- Only 2 of the 6 repositories were actually correct. The other 4 repositories were unrelated/incorrect.
- After further interaction, Rovo now reports the correct 59 commits across 2 repositories.
The major concern is that Rovo is not reliably deriving repository information from the actual commit data. Instead, it appears to be hallucinating repository names and presenting them as factual information.
This is especially problematic because the agent is intended to generate technical implementation documentation. Incorrect repository identification can result in completely inaccurate documentation about impacted systems and components.
Expected: Rovo should identify repositories strictly from the actual commits associated with the Jira ticket and should not infer or fabricate repositories when the underlying data is unavailable.
Current: Commit count eventually became correct (59), but repository identification showed significant hallucination, with 4 out of 6 initially reported repositories being incorrect.
