Using Xray for test management? Here are some tips about integrating automatic tests with Xray

[#1]

Test automation typically requires a significant investment. But even with the best intentions and talented engineers on board, things can sometimes go sideways. When you set up an automated test suite without integrating it with Jira, you create a silo. Project managers often lose sight of the status of regression tests or the development of automated tests. That's why it's essential to integrate your automated tests with the tools that manage work. Additionally, if you cannot utilize the results from your automated tests for traceability reporting, it can lead to unnecessary manual testing and a significant amount of wasted effort.

I've been working with teams to manage requirements and tests using Jira and Xray for many years. In this article, I will discuss this aspect of achieving seamless integration with your test automation suite: how to connect your Xray tests to their corresponding automated test implementations. Specifically, I will explore the question of which comes first: the Xray test or the automated test code?

To create an integrated automated testing solution, you need two key components:

1. Every automated test is represented by exactly one Xray test. The Xray test serves as a proxy that links to stories and requirements. The Xray test creates the test coverage for those Jira work items. It also has an important role, which is to flag that the test exists. This article explains how to establish this link.
2. Execution of the automated test pipeline imports results into Xray, creating test executions and test runs. The import of results to Xray is typically performed only for certain pipelines. For example- We may choose to execute the test pipeline on each commit and on every merge to master. Importing the results to Jira only when merge to master occurs. I will not cover this topic here; see the additional reading links at the bottom for more information.

How do we pair the automated tests with their Xray tests?

By default, the Test Key is the element that links any imported test execution with an Xray test. If the key is not part of the import, there is a fallback:

• For tests labeled as 'Generic,' they'll get matched up using the generic definition field in Xray.
• For other tests, like manual ones, the import will match them based on the Summary field.

Reference: Xray documentation

Here is an example of a Pytest test, annotated to link with the Xray test key:

import pytest


@pytest.mark.xray("DXJ-2")
def test_plus_2():
assert True

Alternatively, here is how Xray can link to the coded test via the class name:

The pairing can be established via two flows:

1. Start with Xray: Begin by creating the Xray test and connect it with the automated test implementation at a later stage. Annotate the test code with the test key or with the test definition. When the pipeline brings results into Xray, it links the test runs to the original tests.
2. Start by creating the automated test. When the pipeline brings in the first round of results into Xray, it'll kick off the creation of a new test case right away. After that, any future import will connect test runs to the original Xray test created during the first import. One notable feature is that the pipeline lets you import links to requirement items in Jira. If you're a test developer working on a test for specific requirements, you can annotate the test with the related Jira work item right in your test code. This way, when the pipeline creates the test case, it'll automatically link up with the corresponding Jira item.

When is it better to start by creating the Xray test?

The optimal flow depends on your use case.

Starting with Xray is a great way to easily integrate test implementation work into your work planning in Jira. If your Scrum team has both developers and QA engineers, you'll usually spot the need for new tests during your refinement meetings. Those tests get created as test work items in Jira/Xray. Once the SQA team starts working on the implementation, they'll shift them to 'in progress.' Plus, they'll annotate the test code to connect it with the Xray test.

The status of the Test implementation appears on the Sprint board, just as it does for any other story.

Another situation where it's better to start with creating the Xray test is when you're switching from manual tests to automated ones. In these cases, it helps to have the automated test case in Jira from the moment you decide to automate it. This way, everyone on the team sees what's coming down the pipeline.

When is it better to first code the test without manually creating the Xray test?

The other approach, where you implement the tests first and then create them in Jira afterward, is helpful for migration or onboarding situations. For example, if you have a set of automated tests developed by teams that are onboarding to Jira or Xray. It's good that the import to Xray can automatically generate the Xray tests.

Another situation that works well for the 'first code, then Xray' approach is when the development of the test is part of the story work, like when developers code unit tests.

And you can also mix and match

Many teams are utilizing both approaches. As long as everyone on the team knows which approach to use in each specific instance, there is no problem.

Integrating your automated tests with Xray eliminates silos and reduces unnecessary manual work. It also lays the groundwork for streamlined release documentation.

Tips and follow-up notes:

1. Here is a playlist with some of our webinars, where we cover the whole integration for specific frameworks.
2. Interested to learn more about SDLC in Atlassian? Read on about it in my series: "Ready for an SDLC shakeup? The good, the bad, and the ugly of managing requirements and tests in Atlassian"