Building a "Veteran Employee" AI Agent in JSM: Looking for Architecture Patterns & Real-World Use Ca

[平野涼]

Hi Atlassian Community,

We are exploring an initiative to create an internal "Veteran Employee" AI Agent integrated with Jira Service Management (JSM).

Our Vision & Use Case:
When team members or agents encounter complex issues or operational incidents in JSM, we want them to be able to ask an AI agent—acting like an experienced senior engineer or operations expert—for guidance. The AI should analyze company-specific runbooks, past incident resolutions, handovers, and documentation to deliver veteran-level responses and troubleshooting steps in a conversational/chatbot style.

What We Are Looking For:

1. Real-World Case Studies / Best Practices: Has your organization built something similar? How has your team configured AI chatbots or custom knowledge agents within JSM?

2. Technical Architecture & Approach:

   • How do you balance native capabilities (e.g., JSM Virtual Service Agent with AI Answers, Rovo Custom Agents) versus custom-built external LLM pipelines (e.g., Forge + A2A Remote Agents / RAG architectures)?

3. Knowledge Ingestion & Governance: What strategies work best for feeding unstructured past ticket history, Confluence runbooks, and handover docs into the AI while ensuring accurate, up-to-date responses?

We would love to hear your insights, recommendations, or lessons learned!

Thanks in advance for sharing your experience!

[Javier Martínez]

Hi Ryo,

This is a very interesting use case. I have been working with similar JSM/Rovo scenarios, and I would probably approach the “Veteran Employee” concept as a knowledge and orchestration problem first, and an LLM problem second.

The main architectural decision I would make is: start native, and introduce an external AI layer only where there is a concrete requirement that the Atlassian-native capabilities cannot satisfy.

1. I would separate the use case into three layers

Layer 1 – Curated operational knowledge

I would avoid feeding years of raw ticket history directly into the agent and expecting it to become the “veteran employee”.

Instead, I would establish a trusted knowledge layer containing things such as:

• validated Confluence runbooks
• known errors and their resolutions
• troubleshooting procedures
• architecture/operational documentation
• post-incident reviews
• service ownership and escalation information

The important distinction here is between knowledge and evidence.

A resolved Jira ticket is evidence that something happened and somebody solved it. It is not necessarily authoritative knowledge. The resolution might have been a workaround, outdated six months later, or very specific to that incident.

2. Use Jira history as contextual evidence

Past incidents are extremely valuable, but I would treat them as a secondary source.

For example, when investigating an incident, the agent could retrieve:

1. the relevant approved runbook,
2. similar historical incidents,
3. previous resolutions,
4. recent changes affecting the service,
5. ownership/escalation information.

Then its answer should make the distinction visible:

According to the current runbook, the recommended procedure is X.
Similar incidents ABC-123 and ABC-456 were resolved by doing Y.

That makes the response much easier for an engineer to trust than a single synthesized answer where the provenance is unclear.

3. Rovo before building a custom RAG platform

For an internal “senior engineer” assistant, I would first prototype this as a Rovo Agent with tightly scoped knowledge.

Rovo can already work with Jira/Confluence knowledge and connected sources while respecting the permissions of the user interacting with the agent. That gives you a significant amount of the retrieval, permission and Atlassian-context layer without having to maintain your own vector database and synchronization pipeline.

I would also consider splitting the “Veteran Employee” into specialized subagents rather than creating one huge agent.

For example:

Veteran Employee
→ Incident Diagnosis
→ Network Operations
→ Database Operations
→ Application Support
→ Change/Deployment Analysis

Each subagent can have narrower instructions and knowledge boundaries.

This normally makes governance and answer quality much easier to control.

4. VSA and Rovo solve slightly different problems

I would not necessarily choose between JSM Virtual Service Agent and Rovo.

I see the Virtual Service Agent primarily as the front door for structured support: answering common questions, identifying intents, collecting information and potentially preventing unnecessary tickets.

The “Veteran Employee” described here sounds more like an agent-assist / engineering investigation use case, where a Rovo Agent is likely the more natural fit.

In practice I could see both coexisting:

Customer/User → VSA → JSM request → Engineer + Veteran Rovo Agent

The VSA handles deflection and intake; the Rovo Agent helps the engineer investigate the difficult cases.

5. When I would introduce Forge / external agents

I would move beyond the native approach when the agent needs capabilities such as:

• querying monitoring/observability platforms
• checking CI/CD systems
• executing diagnostics
• querying CMDBs or operational databases outside Atlassian
• interacting with proprietary internal systems
• performing deterministic calculations or workflows
• using a model/infrastructure that must remain outside Atlassian

At that point Forge can act as part of the integration/orchestration layer, and A2A/Remote Agents become particularly interesting.

One caveat today: I would treat the Remote Agent architecture carefully for production designs because the Jira Remote Agent integration via Forge is currently EAP. It is extremely interesting architecturally, but I would not make a production-critical design dependent on an EAP capability yet.

6. Governance is probably the hardest part

In my experience, the biggest risk in this type of project is not hallucination itself. It is authoritative-looking outdated knowledge.

I would therefore attach lifecycle metadata to operational knowledge:

• service/system
• owner
• last reviewed date
• validity/status
• environment
• version
• escalation path

Then define a simple trust hierarchy, for example:

Approved runbook → Current technical documentation → Recent validated incidents → Historical tickets

The agent instructions should explicitly prefer sources higher in that hierarchy.

For critical operational procedures, I would also instruct the agent to say when it cannot find sufficiently authoritative information rather than trying to infer a solution.

7. Close the knowledge loop

This is where I think the concept can become genuinely valuable.

After resolving an incident:

Incident resolved → resolution reviewed → reusable knowledge identified → runbook/KB updated → agent immediately benefits from it

That turns the “Veteran Employee” from a chatbot trained on historical tickets into a continuously maintained representation of the team's operational knowledge.

If I were implementing this, I would therefore start with a relatively small pilot: one service, its Confluence runbooks and a curated set of high-quality historical incidents.

Measure answer correctness, source relevance and engineer acceptance before expanding the knowledge scope.

The temptation with these projects is usually to ingest everything first. I would do almost the opposite: start with a small amount of trusted knowledge and expand only when retrieval quality is measurable.

Very interesting use case — I would be interested to hear what direction you eventually take.

[Dave Rosenlund _Trundl_]

@Javier Martínez put a lot of thought into his answer, and that is an interesting approach. But I think the question both of you should be asking yourselves is what makes a veteran employee a veteran.

I don’t know that we (AI) are in a place yet where we can train an agent to know what our veteran employees know. There is just too much tacit knowledge in the head of a senior employee and that was never written down.

Moreover, there is a lot of instinct in humans. Past experiences that point us in the right direction. AI has no such experience and training it to have it is a monumental task.

I wrote about this in a recent community article: The Unwritten Layer

[Christopher Yen]

Hi @平野涼 ,

In my organization we're currently exploring Atlassian Rovo Service which provides some similarities of what you're looking for, mainly the supervised resolution management function

It does seem to rely heavily on the quality of your knowledgebase but may be worth looking into at least in your sandbox environment. From what I saw in our sandbox once a human agent takes ownership/assigned a work item with the rovo service enabled there will be a runbook that gets drafted and provided for the agent to review where they can either assign it to Rovo to take action, make changes to the plan, or just use the provided plan as reference.