Since October 2025, our QA team has drafted test cases with an AI agent every single day. Our QA Lead, Konrad Koralewski (ISTQB-certified), has created more than 200 test cases this way.
Below is what changed for our testers, where they still don't trust the output, and how that compares with the latest industry reports.
Is it a revolution? Slowly, and unevenly
AI is already normal in QA. Capgemini's World Quality Report 2025-26 found that 89% of organizations are piloting or using GenAI in quality engineering. But only 15% have scaled it across the whole company. The average productivity gain is around 19%, and a third of companies see very little improvement.
What you hear | What the data shows |
|---|
Everyone already uses AI in testing | Most companies are still piloting it |
AI makes QA dramatically faster | Average gain is about 19%; a third see little benefit |
AI-generated test cases are ready to use | They often look right while missing something |
The hard part is choosing a model | Teams struggle most with privacy, integration and reliability |
AI will replace testers | AI takes over repetitive writing; judgment stays with people |
According to the same report, teams struggle most with data privacy (67%), integration complexity (64%) and hallucinations (60%), rather than with choosing a model.
Where AI really helps: test case design
In PractiTest's State of Testing 2026 report, 70% of testing professionals said they use AI to create test cases. A year earlier it was 41%. That makes sense: test cases are structured text with repeatable patterns.
Our team estimates that writing and refining test cases takes roughly 15% of a tester's week. It's important work, but it's also where fatigue starts. Steps get shorter over time, because long test cases are painful to maintain.
For Konrad, the biggest gain is coverage. Missing an edge case is one of the easiest mistakes to make by hand. AI reads the whole requirement and picks up details a person can skip. But he still decides which test cases stay, reviews them, and runs the tests.
What it still gets wrong
- "Almost right" is the new bug
In Stack Overflow's 2025 Developer Survey, 66% of developers said their biggest frustration is AI output that is almost right, but not quite. QA sees the same thing. A test case can look complete and still miss a precondition or check the wrong result. An almost-right test case may be worse than a missing one, because it creates false confidence.
AI can't test what nobody wrote down. If a requirement doesn't say what happens when a payment fails, the generated test cases probably won't cover it. Worse, the model doesn't always ask. Sometimes it guesses, and a guess is easy to miss in review. Writing better requirements turned out to be part of the job.
AI will happily generate ten correct test cases. It won't tell you which area of the product carries the most risk. The same PractiTest report shows the trap: 70% use AI to create tests, but only 19.9% use it to identify risks. The authors call this the "Faster Horse" effect: more tests, not necessarily better quality.
AI also lacks product knowledge and the customer's perspective. Is this button in the right place? Does the feature actually solve the user's problem? Those questions are still ours.
Why copy-paste into a chat tool didn't work for us
Our first experiments looked like everyone else's: copy the requirement from Jira, paste it into a chat, explain the project, copy the result back, reformat, link it. Across dozens of requirements, those small steps ate most of the time saved. It also meant pasting internal requirements into an external tool, which not every security team accepts.
What worked better was bringing AI to where the requirements already live. Rovo can read the Jira work item, its linked items and project context directly, so the context is complete without the tester assembling it.
How we do it in our own setup
We build a test management app, so we use our own Rovo agent. A tester opens it on a requirement and asks something like "Prepare test cases for ATM-8". It proposes test cases with full steps: action, test data and expected result.
A typical session: the agent proposed six test cases for a new feature. Five were fine. The sixth wasn't what we needed, so we asked for a negative path instead and got a better one a few seconds later. After confirmation, the test cases were created as regular Jira work items and linked to the requirement.
We also decided the agent shouldn't force every possible edge case every time, because the tester usually knows the context better. That's a deliberate human-in-the-loop moment.
If your team is skeptical, start small
- Pick one workflow. Generate test cases for one new feature and compare them with what your team would have written.
- Treat every output as a draft. Accept, reject or fix each test case, and keep a healthy dose of skepticism.
- Write better requirements. Detailed descriptions and linked work items give AI the context it needs.
A year in, AI writes our first drafts, catches edge cases we used to miss and brings order to messy documentation. People still decide what's worth testing and whether a feature is ready to ship.
Over to you
How does your team review AI-generated test cases? Do you track how many you reject or rewrite? Has AI changed how your team writes requirements?
We'd love to compare notes in the comments.
The agent mentioned above is part of Appsvio Test Management on the Atlassian Marketplace.
