Rovo JSM helpdesk agent behaves inconsistently — how do I structure instructions for reliable troubl

[Prabakaran Prabu]

I've built a Rovo agent for our internal IT Service Desk (JSM Cloud). Users interact with it in Slack and on the help center portal. The intended flow is: greet → check for known outages → troubleshoot using our Confluence IT knowledge base → if unresolved or the user asks for a human, create a ticket via the "Raise a request" skill → set expectations (SLA) → ask a quick CSAT question → close.

The problem is inconsistent behavior across conversations. Common symptoms:

• Sometimes it asks the CSAT/feedback question before creating the ticket instead of after.
• Occasionally it calls skills I don't want it to use, or attempts to call a skill more than once (risking duplicate tickets).
• Sometimes it skips the structured description format I require, or leaves troubleshooting context out of the ticket.
• Ordering of steps (create ticket → set expectations → CSAT) isn't reliably followed.

My current instruction set is very long (single large prompt, ~25 sections) and I've tried to fix each failure by adding more explicit rules, forbidden phrases, and repeated "NEVER do X" warnings. It hasn't made behavior more consistent.

Questions:

1. What's the recommended way to structure instructions for a multi-step JSM support agent — core instructions vs. subagents? Should the "troubleshoot" phase and the "create ticket" phase be separate subagents?
2. Is enforcing strict step ordering (e.g. "always create the ticket before asking for feedback") something instructions can do reliably, or should parts of this be handled by JSM automation instead?
3. How should I control which skills the agent can call? Is it better to remove skills at the agent config level than to forbid them in the prompt?
4. Any best practices for the Slack surface specifically, and for keeping the "Raise a request" call to exactly once?
5. General advice on debugging/evaluating agent responses when behavior varies run-to-run?

Any example instruction patterns for a JSM self-service agent would be hugely appreciated. Thanks!

[John D Patton]

Gabriela and Arek hit the nail on the head: bloating a single prompt with "NEVER do X" rules adds cognitive noise rather than control, and multi-step execution order is a job for deterministic workflow engines, not probabilistic LLMs.

To bridge the gap between Rovo's flexibility and JSM's control, you can implement a State-Tracked Data Gathering Pattern. This approach uses the agent strictly for conversational discovery and relies on Jira custom fields to lock down the execution sequence.

1. Move the "Sequence" into a Custom Field

Instead of expecting the agent to remember where it is in the lifecycle (greet $\rightarrow$ troubleshoot $\rightarrow$ escalate $\rightarrow$ feedback), offload that state tracking directly to the JSM ticket or session context.

• The Blueprint: Create a hidden custom field in JSM called Agent Interaction State with values like Troubleshooting, Pending Escalation, and Ticket Created.

• The Execution: When the user transitions from troubleshooting to needing a human, the agent's only write-action is to populate the required description fields and set this state field to Pending Escalation.

2. Let JSM Automation Drive the Gatekeeping

By shifting the state tracking to Jira, you prevent duplicate actions and forced ordering failures through native, deterministic JSM automation rules.

• Enforcing "Exactly Once" Ticket Creation: Set a JSM automation rule that triggers when the Agent Interaction State changes to Pending Escalation. The rule handles the formal ticket creation, stamps the SLA expectations, and instantly flips the state field to Ticket Created. Because the agent isn't executing the creation tool directly, it cannot accidentally call it twice or trigger duplicate tickets.

• Decoupling the CSAT: As Arek suggested, completely decouple the feedback loop from the agent prompt. Use JSM automation to send the CSAT survey via the Portal or Slack only when the native ticket status transitions to Resolved or Closed.

3. Channel Isolation (Portal vs. Slack)

Because user behavior and contextual noise vary heavily between Slack and the Help Center Portal, consider splitting them into two distinct Parent Agents, even if they draw from the same knowledge base.

• For the Slack Surface: Keep the action space as narrow as possible. Slack conversations invite conversational detours, which cause bloated prompts to misfire. By removing the ticket-creation or feedback tools entirely from the Slack-facing agent configuration and limiting its scope purely to knowledge retrieval, you remove any architectural opportunity for the model to jump steps out of order.

By treating Rovo strictly as your Data Gathering Engine and JSM as your Workflow Execution Engine, you eliminate the sequencing burden from the prompt entirely.

[Arkadiusz Wroblewski]

Hello and welcome to Atlassian community @Prabakaran Prabu 

So first... Longer instructions mean there's more possibility of failure.

On the other hand...

Keep CSAT at the 'Done' stage and let SLAs run automatically via background conditions. On Slack, isolate the "Raise a request" tool to the escalation subagent to leverage the native confirmation screen, and ensure no other Assist or VSA apps conflict in that channel.

For troubleshooting, use Rovo's Debug response, evaluation datasets, and transcript reviews. Also, remember that an agent invoked via Automation cannot run tools directly, it only returns text via {{agentResponse}} for your automation rule to handle.

Best,

Arek 🤠

[Gabriela]

The 25-section prompt is the root of it. One agent with a long rulebook stays probabilistic. Each extra forbidden-phrase line adds noise without adding control, and the wrong-order and duplicate-call symptoms are what an overloaded agent does.

Split it. Rovo has native subagents now (the feature that used to be called Scenarios, renamed at Team '26), built in Studio under the parent's Details tab, each with its own instructions, tools and knowledge. Atlassian's own steer is that breaking a bloated agent into focused helpers keeps each one reliable, and to keep each under about five tools. So a troubleshoot subagent and a raise-ticket subagent, yes.

Subagents won't buy you strict ordering though. Delegation is the parent model deciding what to call, in parallel, a step at a time, so there's no builder-defined "do A then B" guarantee in the agent layer at all. Which means anything that has to happen in a fixed order, or exactly once, belongs in JSM automation instead. That part is deterministic. Set the SLA on create, fire the CSAT on resolution, dedupe so a retry can't open a second ticket. The model shouldn't be holding the sequence in its head.

A wrinkle if you go the automation route: an agent invoked inside an automation rule can't run create, update, delete or trigger actions. So the ticket creation runs as the rule's own action, and the agent's job stops at gathering what the ticket needs.

On skills, there's no allow or deny list today, Atlassian says permission control for write actions isn't supported. You limit what it can reach by which tools you attach and how sharply you name and describe them, nothing else. So a skill it shouldn't touch shouldn't be attached in the first place.