nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Testing Rovo agents with test account

Myles Boyd

I am a JSM User/Admin and Atlassian site admin.

I have created a Support rovo agent, linked to both our customer and support KBs. When I test the agent, it uses my details so responds with support KBs.

We use SSO all the way through, so logging in with a different account is not easy.
I don't want to keep pestering staff and asking them.

Has anyone found an "easy" way test the agent as a normal customer, and not use inherited User rights?

Accepted answers

Sami Shaik

@Myles Boyd ,

The reason it answers you with internal articles is not the test account, it is the agent’s identity setting. By default an agent runs as User’s account, and Atlassian’s agent identity page is explicit: “If the agent adopts a user’s account, then the agent can access whatever that user can.” Test it as a site admin and it reads the support KB, because you can.

Fix the agent, not the tester. In Studio, open the agent and change its identity to Agent’s account. It then gets its own account “managed by the organization, space, or app admins (just like a user)”. Grant that account the customer KB space only and nothing else. Now the agent answers from the customer KB whoever is asking, including you, and your admin rights stop leaking into its answers. Per Understand Rovo agent accounts, it operates “within whatever narrow access it has been granted, rather than the full access of a specific user”. That is also the version you want in production: a customer-facing agent that inherits each employee’s permissions is a disclosure risk waiting for the right question.

Then test it the way a customer sees it. Two routes that do not need a colleague:

A portal-only customer account. Create a customer in the JSM space with an address outside your verified domains (a personal mailbox works). Portal-only accounts are not managed by your organization, so your SSO policy does not apply to them. Sign in to the portal in a private window and ask the agent there.
The agent’s own view. With the identity on Agent’s account, your own test in Studio already shows what the agent can read, since it no longer borrows your access. Ask it a question only the support KB can answer; the right result is “I don’t have that”.

One check before you switch: an agent on its own account has no access until an admin grants it, so confirm it can read the customer KB after the change, or it will answer nothing at all.

All comments

Sami Shaik

@Myles Boyd ,

Then test it the way a customer sees it. Two routes that do not need a colleague:

A portal-only customer account. Create a customer in the JSM space with an address outside your verified domains (a personal mailbox works). Portal-only accounts are not managed by your organization, so your SSO policy does not apply to them. Sign in to the portal in a private window and ask the agent there.
The agent’s own view. With the identity on Agent’s account, your own test in Studio already shows what the agent can read, since it no longer borrows your access. Ask it a question only the support KB can answer; the right result is “I don’t have that”.

One check before you switch: an agent on its own account has no access until an admin grants it, so confirm it can read the customer KB after the change, or it will answer nothing at all.

Arkadiusz Wroblewski

Hello @Myles Boyd

Just one additional point to consider when testing this.

Atlassian currently supports Rovo agents in JSM portals only for users with Atlassian accounts. A portal-only customer account therefore wouldn't be suitable for testing your Rovo agent.

I'd recommend using a separate, non-admin Atlassian account with permissions matching a typical customer, ideally in a private browser session.

Switching to the Agent's account changes which permissions the agent uses, but doesn't simulate the permissions of an individual customer. That distinction is important when testing knowledge base access.

Atlassian explains limitation here:
https://support.atlassian.com/jira-service-management-cloud/docs/set-up-your-virtual-agent-in-jira-service-management/

Sami Shaik

@Arkadiusz Wroblewski , thank you, and you are right. I re-read the Set up your virtual service agent page this morning and it says it plainly: “Rovo agents are currently only available in Jira Service Management portals for Atlassian Accounts.” So the portal-only customer route in my answer does not work for a Rovo agent on the portal. Withdrawn.

@Myles Boyd ,

the corrected version of the test step:

Keep the identity change. Switching the agent to Agent’s account and scoping it to the customer KB is still what stops your admin access leaking into its answers. That part is about what the agent can read.
Test with a real Atlassian account that is not under your SSO policy. Arkadiusz’s suggestion is the right one: a non-admin Atlassian account with customer-level access, in a private window. If every account on your verified domains is SSO-enforced, create the test identity on an address outside those domains (a personal mailbox registered as a full Atlassian account, not a portal-only customer) and add it as a customer of the space. Your authentication policy applies to managed accounts, so an unmanaged account on an unverified domain signs in with a password. That part is about what the tester can see, and the two halves together give you the customer’s view.

Reference: https://support.atlassian.com/jira-service-management-cloud/docs/set-up-your-virtual-agent-in-jira-service-management/

Welcome!

It looks like you're new here. Sign in or register to get started.