Rovo Agent hallucinating runbook summaries — fabricates steps not present on source Confluence page

[Akash Jha]

I'm using a Rovo Agent automation rule to retrieve Runbook Summaries from child Confluence pages linked under a parent page.

The agent correctly identifies and navigates to the child runbook links. However, the generated summaries contain fabricated content — steps, commands, and details that do NOT appear on the actual Confluence page.

Observations

Case 1: Summary is 100% hallucinated (Critical)

For the "Astro DC Disconnects" runbook, NONE of the steps in the agent's summary exist on the page:

| Agent's summary step | On actual page? |
|---|---|
| HugePages / `grep HugePages /proc/meminfo` | ❌ Fabricated |
| JVM Metaspace / classloader leaks | ❌ Fabricated |
| Java Flight Recorder / heap dump | ❌ Fabricated |
| Restart astro-2 | ❌ Fabricated |

**What's actually on the page:**
- DC disconnect ratio < 0.96 threshold
- `service:astro AND msg:"socket connection closed"` Kibana query
- Dashboards grouped by Node/Pod/AccountID
- Contacts: David King (SRE), Sean Hagan (DE)

The agent appears to have read the page content (it dumps it below the summary), then ignored it and generated a generic memory troubleshooting guide instead.

Case 2: Mix of real content + hallucinated commands

For the Kafka partition depth runbook:

| Detail | On actual page? | In agent summary? |
|---|---|---|
| 100,000 queue depth threshold | ✅ Yes | ❌ Not mentioned |
| Kibana query `msg:"finished processing stimulus"` | ✅ Yes | ✅ Yes |
| Saved viz `sre_kafka_impact_customer_list` | ✅ Yes | ❌ Missing |
| `kubectl rollout restart sts dejavu` | ❌ NOT on page | ✅ Included |
| `kubectl scale sts dejavu --replicas=30` | ❌ NOT on page | ✅ Included |
| `kafka-consumer-groups.sh` command | ❌ NOT on page | ✅ Included |

Environment
- Atlassian Cloud (Jira + Confluence)
- Rovo Agent invoked via Jira Automation rule

Question
1. Is there a way to constrain the Rovo Agent to ONLY summarize content that exists on the page (no generation/inference)?
2. Are there known prompt patterns to prevent hallucination when summarizing Confluence pages?
3. Is this a known limitation of Rovo's current Confluence page reading capability?

[Dr Valeri Colon _Connect Centric_]

Welcome to the Community, @Akash Jha.

What you're describing goes beyond normal summarization drift. If the agent is introducing commands, thresholds, or remediation steps that don't exist on the source page, that's a grounding issue rather than a simple prompt issue.

Prompts can reduce this risk, but there is currently no way to guarantee Rovo will only produce extractive summaries. Given the operational nature of runbooks, I'd raise this with Atlassian Support as a potential product limitation or defect.

[Stephen_Lugton]

Hi @Akash Jha 

To answer your first question, it is possible to update the agent prompt with something like:

CRITICAL INSTRUCTIONS:
1. You are a strictly factual assistant. Rely ONLY on the explicitly provided data.
2. Do not infer, speculate, extrapolate, or assume facts not directly contained in the source material.
3. If the provided data does not contain the answer, you must respond EXACTLY with: "Insufficient information."
4. Do not attempt to guess or synthesize an answer if the data is incomplete.
5. Base every claim solely on verifiable text within the context.

or

Prioritize accuracy over helpfulness.

If you don't have reliable information on something, say "I don't have reliable information on this" instead of guessing or extrapolating.

If your knowledge might be outdated (especially for anything after January 2025), explicitly flag it: "My information is from [date]—this may have changed."

If you're uncertain about a fact, statistic, or claim, say so clearly: "I'm not confident about this, but based on what I know..."

If something requires current data you don't have, tell me: "This needs up-to-date information. Let me search for that."

Don't fill gaps with plausible-sounding answers. Don't smooth over uncertainty with confident language. Don't assume I want an answer more than I want the truth.

If you need to guess or reason from incomplete information, explicitly separate what you know from what you're inferring.

Treat "I don't know" as a valid and valuable response. I'd rather hear that than confidently wrong information.

[Philipp Sendek _catworkx_]

Hi @Akash Jha ,

while I'm confident that our Rovo Champions in the company might be able to share their experience and make suggestions, I saw that you're on an Enterprise plan and hence would recommend that you create a ticket with the Atlassian support so that they can have a look at your site and troubleshoot this properly with you.

You can create a support ticket here: https://getsupport.atlassian.com/

Greetings
Philipp