Evaluation Feature Failing for Valid Agent Responses

[Jagruti Shinde]

Hi everyone,
I’ve recently started experimenting with the newly launched Evaluation feature by Atlassian and have encountered an issue.
I created an agent and tested it through normal conversation. In this case, the agent responds correctly and behaves as expected for the given inputs.
However, when I use the same inputs via a CSV file in the Evaluation feature, all the test cases are marked as failed, even though the responses appear to be valid.
Has anyone faced a similar issue or knows what might be causing this behavior? Any guidance or suggestions would be greatly appreciated.
Thanks in advance!

[Rebekka Heilmann _viadee_]

Hi @Jagruti Shinde - welcome to the Community,

can you share the setup of the CSV and the Evaluation itself? There are a few settings that influence how the evaluations show up.

"Failed" usually means that Rovo deviated from the expected response. But as that evaluation happens via LLM, they are not always accurate. 

Evaluations best work with classification Agents that give a clear Yes or No answer (or whatever other classification you use).