Agentic Test Execution for Xray: Questions to Ask Before You Start

[#1]

It's Thursday of release week. Your Xray Test Execution issue has 80 manual tests; you have two testers available, and experience says it takes two full days, including review. The release is planned for Monday. You've been here before, and so has every QA team working in Jira: caught between manual execution that doesn't scale and script-based automation that isn't practical yet for features still evolving sprint after sprint.

 Over the past year, a new option has emerged to fill this gap: agentic test execution. An AI agent reads your existing manual and Gherkin test cases exactly as they are written in Xray (no rewriting and no scripting), drives the application under test, and returns step-by-step verdicts with evidence.

 Full disclosure before we go further: I am a co-founder of Smartesting, the company behind Lynqa for Xray  on the Atlassian Marketplace. This article walks through the decision framework we use with QA teams to answer a harder question than whether the technology works: is agentic test execution actually a fit for your context? 

The three questions (Applicability, Reliability, and Value) hold up whatever tool you evaluate, and "not yet" is a perfectly valid answer. But first, let’s look at what agentic execution means inside Jira and Xray.

Agentic test execution inside Jira and Xray

The principle is simple to state: instead of a human following the steps of a manual test, or a script replaying hard-coded selectors, an AI agent interprets the natural-language instructions of your test case the way a manual tester would: it understands the intent, performs the actions in the browser, observes the application's responses, and issues a verdict per step.

 Concretely, in a Jira + Xray setup with Lynqa, the workflow looks like this:

1. Select Manual or Gherkin/Cucumber test cases from an Xray Test Execution, unmodified.
2. Choose the target test environment and launch agentic execution.
3. The agent executes: it reads each step's instructions, performs the actions, and validates expected outcomes, capturing screenshots and evidence as it goes. Runs can be parallelized to absorb peak load.
4. Review in Jira: the team examines the results, focusing on the behaviors the agent flagged as unexpected, then saves the results and findings in Xray.
5. Results land back in Xray dashboards (requirement coverage, defect traceability, release readiness), exactly where the rest of your reporting already lives.

The following two screenshots show what it looks like in Xray: 1) the button to launch agent-based execution with Lynqa, and 2) the execution context configuration.

Figure 1 - Launching agentic test execution with Lynqa from an Xray Test Execution issue

Figure 2 - Configuring the agentic execution context.

Two things matter in this workflow. First, nothing about your test repository changes: the tests stay in Xray, in natural language, owned by the QA team. Second, the human stays in the decision seat: the agent executes and documents; the tester decides.

 Now, the three questions.

Applicability: Is your test suite a fit?

Agentic execution is not a universal replacement for either manual testing or scripted automation. It shines on a specific but very common profile of tests:

• Functional UI tests on web applications, especially interfaces that are still evolving
• Manual test cases written in natural language, with a clear intent and expected results
• Gherkin/Cucumber scenarios that were never automated (or whose automation you dread maintaining)
• Acceptance criteria of user stories: the sprint-validation tests that always land at the worst moment

The differentiator versus scripted automation is tolerance to change. A script breaks when a selector moves; an agent interprets the objective ("add the item to the cart and verify the total updates") and adapts to a redesigned interface, as a human tester would. This means you don't need a functionally stable application to start, which is precisely where traditional automation is not yet worth its maintenance cost.

The right mental model is complementarity, not replacement: agentic execution covers the moving parts of the product, while your existing scripted suites keep covering stable regression. Xray's own positioning on this combination is the same: agentic execution for immediate coverage, AI-assisted script generation when scenarios stabilize and are worth automating long-term.

 Where it is not a fit today: tests requiring physical hardware, native mobile or desktop applications (web only for now), and tests whose data or preconditions the agent cannot access.

Reliability and trust: Can you verify the verdicts?

This is the question that should worry you most, and rightly so. A test execution report is only useful if you can trust the verdicts, and "trust" in an AI context cannot mean blind faith. It has to mean verifiability.

Two mechanisms matter here.

Measured accuracy. Ask any vendor for their benchmark methodology. For Lynqa, we publish an assessment framework measuring true accuracy (the verdict is counted correct only when every step verdict and the outcome are right), currently above 92% across 1,247 scenarios on 215 web applications. Whatever the tool, insist on true accuracy rather than headline success rates: a wrong verdict that looks confident is worse than a failed run.

Evidence per step. Every executed step should return the actions performed, what the agent observed, the verdict, the reasoning behind it, and screenshots. That's what turns a review from "re-run it manually to check" into "scan the evidence in minutes." In the Xray workflow, this evidence is attached to the execution results, so your traceability chain (requirement → test → execution → defect) stays intact.

Here is a screenshot of the agent-based execution report for the test performed using the execution configuration shown in Figures 1 and 2.

Figure 3 - Step-by-step Lynqa execution results and screenshots inside Jira

And because the agent operates on your applications, it should only ever touch test environments, never production, and your security team will want the usual guarantees (for Lynqa: SOC 2 Type 2 and a self-managed option for on-premises deployment).

Value: What do you actually gain?

Forget generic "10x" claims; the value equation is specific to your context. Three levers are worth quantifying:

• Execution capacity. A representative case from our deployments: a regression cycle that took two testers two days compressed to about two hours of agent execution plus human review, thanks to parallelization. Your mileage depends on suite size and review discipline: measure it, don't assume it.
• Avoided automation cost. Every test the agent executes from natural language is a script you don't write and, more importantly, don't maintain through UI changes.
• Where the freed time goes. The point is not fewer testers; it's testers doing exploratory testing, risk analysis, and test design instead of stepping through the same flows for the nth time.

 If you run a pilot, track three metrics across several cycles: hours saved per cycle, including review time; defect-detection parity with human execution (does the agent catch what your testers catch?), and actual reuse of the freed time. The third one is the honest test of whether the gain is real for your team.

A quick self-assessment checklist

You're a good candidate for a pilot if several of these ring true:

• You have functional UI tests (manual or Gherkin) in Xray today;
• Manual execution delays are blocking incremental delivery or compressing release windows;
• Your features and interfaces evolve regularly (which requires technical expertise for maintenance, even with AI);
• Your test cases express clear intent and expected outcomes;
• Test data and environments are available for an agent performing the tests.

If most boxes are unchecked (for example, your tests are terse one-liners only their author understands), the first step isn't an agent; it's improving test expression. That investment pays off regardless of how the tests are performed.

How to start: pick a small, representative set: common paths, a few edge cases, one end-to-end scenario. Run it over several cycles, measure the three pilot metrics, then decide whether to widen the scope. Don't start with your full regression suite.

Wrapping up

Agentic test execution won't replace your testers or your automation strategy. It offers round-the-clock availability to run as many manual tests as you like in parallel, and it does so from the test cases you already have in Xray.
Ask the three questions: Is my test suite a fit? Can I verify the verdicts? And what do I measurably gain? If the answers line up, a pilot is cheap to run and quick to evaluate.

The app is available on the Atlassian Marketplace: Lynqa for Xray - Agentic Test Execution.

 Over to you: how is your team handling the manual-execution crunch at release time: more people, more automation, or rethinking the process? I'd be curious to hear what's working (and what isn't) in the comments.