How are teams comparing the quality of different AI coding agents in Jira?

[Chris May]

Teams can now use several different AI coding agents alongside Jira, but I’m interested in what happens after the agent completes its work.

How are teams determining whether one agent performs better than another?

For example:

• Was the output accepted without changes?

• Did it require minor editing or substantial rework?

• How frequently did the agent fail to complete the task?

• Are results compared by project or type of work?

• Or is agent selection still mainly based on individual developer preference?

I have built a small Forge prototype under Aigeeza to explore this question. It allows someone reviewing a Jira task to record the agent used and classify the result as accepted, minor edits, major rework or failed. The results are then compared across projects and time periods.

Before developing it further, I would appreciate some honest practitioner opinions:

1. Does your team regularly use more than one AI coding agent?

2. Would this kind of comparison help you choose between them?

3. Is a four-level human assessment too simplistic?

4. What evidence would you need before trusting the resulting scores?

I’m not sharing an installation link at this stage. I’m primarily trying to establish whether this is a genuine problem for other Jira teams or merely something that interests me.

For transparency, AgentScore is a prototype I have built as part of my business projects on Aigeeza.

[Ben Spillane]

Hi @Chris May , 

You raise a great question. Developers in my workplace often have lively conversations over which coding agent produces the best results. Being a software engineer myself, I've definitely been part of such conversations; they often morph into debates.

From my experience there's plenty of AI coding stories that fly round the room, each one as passionate as the last. The stories usually focus on what was built, how fast it was to produce and how simple it was to achieve (compared to traditional coding). 

I've been using various coding agents for quite some time now, including Gemini, GitHub Copilot (differing underlying models), Claude and Rovo Dev. And you know what, they can all code faster than me! The question is though, what's your measure of an agents performance?

I'm currently exploring agentic team concepts, such as workflow design, cross agent compatibility, agent automations, human->agent role mapping, and so on. The questions you're asking are reasonable, however, I'll answer each of your (quality scoring) questions with a question (or several):

• Was the output accepted without changes?

  • Were the requirements and contextual input supplied without mistakes or omissions?

  • Does the team have pre-existing review steps within the workflow? If yes, what's the gated criteria for a pass/fail - what's the threshold for acceptance without change?

• Did it require minor editing or substantial rework?

  • To what level and criteria are the coded changes evaluated?
  • What constitutes a minor edit vs substantial rework and do both of these block the shipment of the changes?
  • Were objective and measurable acceptance criteria provided as part of the input to facilitate a suitable evaluation?

• How frequently did the agent fail to complete the task?

  • What's the threshold and definition of failure?
  • Is failure detection already part of the workflow?
  • Are workflow opportunities provided to prevent or remedy potential task completion failures?

• Are results compared by project or type of work?

  • Are the results objectively comparable?
  • Assuming acceptance criteria are provided and met across all solutions, how are ties broken?

I'll also provide some thoughts on your other questions:

• Does your team regularly use more than one AI coding agent?
  • I'm carrying out some thorough real world trials incorporating numerous AI coding agents. 
  • It's unlikely an organisation will invest in multiple AI coding platforms. (This doesn't account for developers having personal coding agent subscriptions in addition to company "standard" one.)
    
    
• Would this kind of comparison help you choose between them?
  • I've found corporate organisation to be heavily influenced by compliance and governance related comparisons, as well as costs and platform features (e.g. enterprise features, system integration capabilities, authentication options etc).
• Is a four-level human assessment too simplistic?
  • Perhaps. But again, it depends entirely on what's the objective measure of the assessment.

• What evidence would you need before trusting the resulting scores?
  • Resulting scores need to have clear justification and documented means of calculation. For example, if "major rework" of a agent's solution was required, then assessment evidence would be provided to justify the rework claim.

This is a fascinating subject and I'm always keen to hear how others in the industry are wrestling with our new AI world.

Thanks for sharing.

Also, keep an eye out on the forums as I'm planning to write a few articles in the coming weeks that explore the questions I posed earlier. I'll dig into answers to those questions while focusing on agentic / AI native team concepts and setup in Jira.