How are we validating Rovo Agent data boundaries before production rollout?

Prashanth

Hi All,

While experimenting with a Rovo Agent, I scoped its knowledge to specific Jira projects and Confluence spaces by disabling "All Projects" and explicitly selecting only the sources I wanted the agent to use.

As an initial validation test, I asked the agent what projects it could access, and it correctly returned only the scoped projects.

That gave me some confidence, but before considering broader adoption, I'd like to better understand how others are validating data isolation and access boundaries in real-world deployments.

A few questions I'm exploring:

1. Are there audit logs, execution traces, or admin views that show which Jira projects or Confluence spaces were queried during agent execution?

2. If a new project is created later, does the agent remain limited to the originally scoped projects, or are there scenarios where additional content becomes discoverable?

3. For external connectors such as Google Drive or Slack, how are you validating the interaction between native source permissions and Rovo Agent scoping?

4. What negative testing approaches have you found effective? For example, intentionally prompting for content outside the approved scope to verify boundaries are being enforced.

5. For those already running Rovo Agents in production, what did your validation process look like before making agents available to end users?

My main objective is ensuring that the configured knowledge scope and underlying permissions behave exactly as expected before wider rollout.

Interested in hearing how platform teams are approaching this.

Nathalia Carvalho

Hi,

We're still in the evaluation phase ourselves, but our approach has been to assume that scoping and permissions are separate controls that both need validation.

For validation, we've focused primarily on negative testing:

• Ask the agent about projects and spaces that are intentionally out of scope.
• Reference specific issue keys, page titles, or known content outside the approved scope.
• Test with different user accounts to confirm that user permissions are still being respected.

For newly created projects, I would recommend testing immediately after creation rather than assuming the original scope remains unchanged. We treat any change to projects, spaces, or connectors as something that requires revalidation.

Regarding auditability, one of the gaps we've been looking at is visibility into exactly which sources were consulted during agent execution. If anyone has found a reliable way to trace project, space, or connector access during a Rovo interaction, I'd be interested as well.

For external connectors, our expectation is that native source permissions remain the primary security boundary, with Rovo scoping acting as an additional layer rather than a replacement for access controls.

Before broader rollout, we've been using a combination of:

• Positive tests (content that should be accessible)
• Negative tests (content that should not be accessible)
• Multiple user personas with different permission levels

This has given us more confidence than relying solely on the configured scope settings.

I'd be interested to hear how others are approaching validation in production environments.