How are we validating Rovo Agent data boundaries before production rollout?

[Prashanth]

Hi All,

While experimenting with a Rovo Agent, I scoped its knowledge to specific Jira projects and Confluence spaces by disabling "All Projects" and explicitly selecting only the sources I wanted the agent to use.

As an initial validation test, I asked the agent what projects it could access, and it correctly returned only the scoped projects.

That gave me some confidence, but before considering broader adoption, I'd like to better understand how others are validating data isolation and access boundaries in real-world deployments.

A few questions I'm exploring:

1. Are there audit logs, execution traces, or admin views that show which Jira projects or Confluence spaces were queried during agent execution?

2. If a new project is created later, does the agent remain limited to the originally scoped projects, or are there scenarios where additional content becomes discoverable?

3. For external connectors such as Google Drive or Slack, how are you validating the interaction between native source permissions and Rovo Agent scoping?

4. What negative testing approaches have you found effective? For example, intentionally prompting for content outside the approved scope to verify boundaries are being enforced.

5. For those already running Rovo Agents in production, what did your validation process look like before making agents available to end users?

My main objective is ensuring that the configured knowledge scope and underlying permissions behave exactly as expected before wider rollout.

Interested in hearing how platform teams are approaching this.

[Nathalia Carvalho]

Hi,

We're still in the evaluation phase ourselves, but our approach has been to assume that scoping and permissions are separate controls that both need validation.

For validation, we've focused primarily on negative testing:

• Ask the agent about projects and spaces that are intentionally out of scope.
• Reference specific issue keys, page titles, or known content outside the approved scope.
• Test with different user accounts to confirm that user permissions are still being respected.

For newly created projects, I would recommend testing immediately after creation rather than assuming the original scope remains unchanged. We treat any change to projects, spaces, or connectors as something that requires revalidation.

Regarding auditability, one of the gaps we've been looking at is visibility into exactly which sources were consulted during agent execution. If anyone has found a reliable way to trace project, space, or connector access during a Rovo interaction, I'd be interested as well.

For external connectors, our expectation is that native source permissions remain the primary security boundary, with Rovo scoping acting as an additional layer rather than a replacement for access controls.

Before broader rollout, we've been using a combination of:

• Positive tests (content that should be accessible)
• Negative tests (content that should not be accessible)
• Multiple user personas with different permission levels

This has given us more confidence than relying solely on the configured scope settings.

I'd be interested to hear how others are approaching validation in production environments.

[Annie Ioceva _Nemetschek Bulgaria_]

We have not conducted many data isolation tests so far, and I would be interested to learn more about this area. I believe it will become an even greater challenge in the future as more business processes rely on automation and AI-driven workflows.

The overall complexity is likely to grow significantly. At some point, the scope and number of interactions may become so large that AI will be the only practical way to perform comprehensive data isolation testing across all scenarios.

[Prashanth]

@Nathalia Carvalho ,Thank you for the response and good that you are approaching a positive, negative and a mixed/neutral approach.

A few follow-ups:

When you test with different user accounts, do you use actual personas with different permission levels, or simulate this another way?

Do you track which prompts failed vs. passed systematically (spreadsheet, something else), or is it more informal?

On the audit trail gap, Rovo has audit logs for admin actions (agent created, chat started, etc.), but I haven't found fine-grained tracing for which specific sources were queried during execution. If anyone finds a reliable way to trace project/space-level access per query, that'd be valuable.
 

Appreciate the practical input.

[Prashanth]

@Annie Ioceva _Nemetschek Bulgaria_  Thank you Annie, Appreciate the honesty.

You're right, this will only get more complex as AI workflows scale. For now, I'm starting with manual negative testing on a small scope, but that won't scale long-term.

The point about AI potentially being the only practical way to test AI at some scale is interesting. Worth keeping an eye on.

[Nathalia Carvalho]

 Thank you for the follow-up questions.

Regarding the user accounts, we use actual personas with different permission levels, such as admin, manager, and standard user profiles. This helps us validate both the quality of the responses and whether the agent respects the access boundaries for each role.

For prompt tracking, we are using a structured spreadsheet where we register the prompt, expected result, actual response, and pass/fail status. This has helped us keep the testing more consistent and easier to review.

About the audit trail, we had a similar finding. We can see admin and operational logs, but we have not yet found a reliable way to trace, at a detailed level, which specific sources, projects, or spaces were queried during each execution. If we identify a good approach for that, I’ll be happy to share it with the group.

Appreciate your comments and insights as well.

[Prashanth]

Great, Thank you @Nathalia Carvalho , for the moment what all you and your team doing is perfect.

On the prompting perspective, I found using XML tags very useful for lengthy prompts like 500,1000 + lines.

I posted a discussion here yesterday, still waiting to hear from other practitioners, have a glance at your convenience.

Have a great day :)