Evaluation Feature Failing for Valid Agent Responses

[Jagruti Shinde]

Hi everyone,
I’ve recently started experimenting with the newly launched Evaluation feature by Atlassian and have encountered an issue.
I created an agent and tested it through normal conversation. In this case, the agent responds correctly and behaves as expected for the given inputs.
However, when I use the same inputs via a CSV file in the Evaluation feature, all the test cases are marked as failed, even though the responses appear to be valid.
Has anyone faced a similar issue or knows what might be causing this behavior? Any guidance or suggestions would be greatly appreciated.
Thanks in advance!

[Rebekka Heilmann _viadee_ · Accepted solution]

Hi @Jagruti Shinde - welcome to the Community,

can you share the setup of the CSV and the Evaluation itself? There are a few settings that influence how the evaluations show up.

"Failed" usually means that Rovo deviated from the expected response. But as that evaluation happens via LLM, they are not always accurate. 

Evaluations best work with classification Agents that give a clear Yes or No answer (or whatever other classification you use).

[Dr Valeri Colon _Connect Centric_]

@Jagruti Shinde a “Failed” result doesn’t necessarily mean the agent response itself is wrong. If you’re using Response Accuracy, an LLM judge compares the response against the expected answer in your CSV. Open a failed case and check the judge’s reasoning. I’d also run the same dataset using Manual Testing, which removes scoring. If those responses look correct, you can narrow the issue to the expected answers/evaluation rather than the agent itself.