We are using Jira Server. Tests and managed by test sets/plans and run via a test execution. In my example; lets say I have a test execution: 'Sprint 1' that has test x with a status: PASSED and then later I have a new test execution: 'Sprint 2' for the same test plan and now test x has a status: 'BLOCKED' or 'TODO'.
I would like to have a dashboard showing the latest test result, however when pulling the test or the test plan it is showing test x with a status of 'PASSED' which is not accurate.
In my original example, if test x is set to 'FAILED' in 'Sprint 2' then the status IS reflected. I can see that test x is 'FAILED' when viewing a dashboard showing either the test or test plan state. This seems inconsistent, is it a bug?
Edit: we are using XRAY plugin as a test management tool
Hello @Jamie Mcdevitt
Welcome to the community!!
The difference in the status of issues on the dashboard could be due to the reason that the gadgets are not set to refresh after a defined duration of time.
This can be done by the owner of the board. Kindly connect with them for this.
Please let me know if you would need the steps for this.
Kindly accept the answer, if this helps resolve the query.
Hi @Gaurav, thank you for your reply however this is not the issue. I have already enabled the gadgets to refresh every 15 minutes but the status remains the same. The issue is not solely with the dashboard, If I open up the testplan the 'Overall Execution Status' reflects the old value. It is specific to the states: 'TODO' and 'BLOCKED'.
If a test case within an OLD test execution has a test result status and later rerun as part of a new test execution, the 'Overall Execution Status' will always choose the older state if the new state is 'TODO' or 'BLOCKED', if any other state is chosen then it will reflect it accurately. This almost seems like its done by design but its frustrating and feels more like a bug.
Hello @Jamie Mcdevitt
It seems there is a status mismatch, and the test execution reflects the OLD run status and not the NEW run status.
In my limited experience, we need to create new Test Plans and add the Test Cases prior to execution. I would suggest that you can try that and see if that helps (if not doing it already :))
@Gaurav per your suggestion, are you saying for each rerun of a test case, we should be creating new test plans every time? Is that not the purpose of the test execution?
To confirm I will go through an example:
Test plan: SERVICE_X_SMOKE_TESTS - Contains 10 test cases
Test Execution: SPRINT_1_SERVICE_X_SMOKE_TESTS - Result: 8 pass 2 fail
Team reviews results, decides they will not fix the 2 failed tests in sprint 1, agrees to block the tests.
Test Execution: SPRINT_2_SERVICE_X_SMOKE_TESTS - Result: 8 pass 2 blocked
Test plan: SERVICE_X_SMOKE_TESTS - Now reflects two test executions linked, clicking into each execution reflects the results correctly but the 'Overall Execution Status' of the test plan reflects 8 passed 2 fail
The test plan is now not reflecting that the team has blocked 2 tests and that in the latest execution linked their result is indeed blocked. Per your suggestion instead of this approach the team needs to create a new test plan EVERY time they wish to run an execution for truly accurate results? This seems counterproductive to how test plans/test executions work and defeats the purpose of linking executions to plans.
Additionally to this, if the result is ALWAYS pass/fail then as I stated before the 'Overall Execution Status' is correct and reflects the delta correctly. It just IGNORES blocked/todo statuses and appears to be a bug?
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events