On my current project, most tickets get their first round of testing from an AI agent. I type /test-ticket ABC-123 and walk away. When I come back, it has read the ticket, read the pull request, opened the designs and logged into a deployed copy of the app. It has also clicked through every acceptance criterion, tried the edge cases nobody wrote down, and drafted a test report with screenshots for me to review.
This isn't a demo. It's how QA works day to day on a regulated financial services product. This post covers what it takes to set up, what the agent does, and where it still needs a human. Spoiler: the setup is mostly things good teams already do.
Start with the boring bit
An agent tests against what it can read. If the ticket is vague, the testing will be too. The practices that make agentic QA work are the ones we've cared about at Red Badger for years:
Small, well-defined tickets in Jira (or similar), with clear acceptance criteria. The ACs help form the agent's test plan.
Every ticket linked to its pull request. We start PR titles with the ticket key, so the agent can find the code from the ticket alone. It also means Jira finds the PR and populates its Pull Request field correctly.
None of this is AI-specific. A human tester benefits from all of it too. The difference is that a human can fill the gaps by asking around, and an agent mostly can't.
Give the agent the same access a tester has
You need an agent harness first. I use Claude Code, but others would do. The harness matters less than what you connect it to.
The team's tools. The agent connects to Jira, Figma and Datadog through MCP servers, and to GitHub through the gh command line tool. How each one is connected doesn't matter much. What matters is that the agent can now read a ticket, a diff, a design or a trace by itself.
The app. For a web app, the Chrome DevTools MCP server lets the agent drive a real browser. It can click, type, resize the window, take screenshots and run JavaScript on the page. For native mobile apps there are equivalent tools.
Where and how to get in. The agent needs to know whether to test a deployed environment or run the app locally, plus test account credentials. We keep test credentials in a local env file the agent reads. They're never real customer accounts, and they're never for production.
Everything the agent finds goes through a person before it reaches the team.
One command: /test-ticket
With that access in place, you could type "test ABC-123 please" and get something useful. You'll get much better results by writing down what a good QA engineer would do, once, and reusing it. Claude Code calls this a skill. It's a markdown file of instructions, and ours is called test-ticket.
A run goes like this:
Gather context. It reads the Jira ticket: user story, ACs, notes and out-of-scope items. It finds the PR by ticket key, checks out the branch, reads the diff, and picks up the environment URL from the PR comments.
Fetch the designs. It pulls every linked Figma frame, including the mobile, empty and error variants.
Choose a strategy. UI changes get tested in the browser on the deployed environment. Backend changes might mean calling APIs, reading logs or running the app locally. Many tickets need a mix.
Test it like a person would. It logs in, finds the feature and works through it. It does this at desktop and mobile widths and in the empty, loaded, loading and error states.
Measure, don't eyeball. For design checks it reads the computed styles off the page (padding, font sizes, colours, border radius) and compares them with the Figma values. It resolves our colour tokens to hex rather than trusting the names to match.
Mark every AC pass or fail. One at a time, with evidence.
Go off-script. It brainstorms edge cases the ticket doesn't mention, then tests them.
Report. More on that below.
The skill also tells it what to do when something's missing, such as an MCP server that isn't connected. In that case it walks the person through setting it up, so anyone on the team can run it.
Two reports, two audiences
The skill's early Jira comments were too thorough and almost unreadable. They were long and technical. In theory a developer could have used them. In practice they were so wordy that even the developers struggled to understand the actual problem. The BA, designer and delivery lead reading the ticket had no chance.
So the skill now produces two reports.
The Jira comment opens with a one-line verdict, such as "All six ACs pass, two differences from the design". Then comes an AC table with pass and fail labels, then a short numbered item per problem. Each problem has a screenshot embedded right next to it, with the Figma frame alongside where there's a design. It doesn't restate the ticket or narrate its process. Rows that passed aren't repeated.
It also doesn't post anything until I've read the working report and said yes. If the problems it found were already there before this change, it uses a second skill, /log-a-bug, to raise them as separate bugs, with plain-English steps anyone can follow to reproduce them.
It doesn't only click buttons
Some scenarios are hard to reach through the UI. A good tester could reach every one of these, but each takes time: hand-editing API responses, changing code to force an error, seeding thousands of records, writing a script to fire off a burst of requests. Under deadline pressure, that work tends to get squeezed. The agent sets it up far faster, so it happens on every ticket that needs it, not just the ones where someone had a spare afternoon. Some real examples from the project:
Faking states the environment can't produce. When a test environment couldn't create the data a ticket needed, the agent rewrote the API responses inside the browser. That let it check the "already signed" and "one party still to sign" versions of a page against the real deployed frontend.
Forcing errors locally. Error states are hard to trigger on demand. It runs the app locally, changes the code to force the failure, checks how the UI handles it, then puts the code back.
Going below the UI. For backend tickets it calls the APIs directly and streams the server logs. For example, it confirmed that a file download really reached the server, not just that a spinner went away.
Checking observability. For tickets that add metrics or tracing, it queries Datadog to confirm the data actually arrives, in the environments that report it.
Testing at volume. For a performance ticket it seeded a test environment with 200,000 records and timed the filters. Nothing had regressed, but it found a sort order that would hit a much slower query plan as data grew.
Testing races. For an upload race condition it fired 100 uploads at once. One green run proves nothing for a race, so it reports how many times it ran the test.
These aren't steps in the skill. They come from giving a capable agent the goal of verifying the ticket and letting it work out how.
Testing the API, not just the UI
Testing through the UI only covers the requests the UI happens to send, but a buggy client or an attacker can send anything. So when a PR adds or changes an endpoint, the agent works out the endpoint's contract and builds a matrix of everything that could go wrong. That includes things like input validation, missing auth, and one client asking for another client's data, etc. It sends every case as a real request and checks the response. It also checks that a rejected request didn't change anything.
That's dozens of requests per endpoint, which is tedious for a person and no trouble for an agent. The Jira comment lists only the failures, each with the command to reproduce it.
How it gets better
The skill is a text file in the repo. If the agent keeps missing a kind of problem, or writes findings up in a way the team doesn't like, I edit the skill. The change gets reviewed like any other, and every teammate who runs /test-ticket picks it up.
Where it goes wrong
It isn't infallible, and it's wrong in the same confident tone it uses when it's right. Some things I've seen:
Closing a bug without testing it. It read the code, saw that the faulty logic had changed, noticed a green end-to-end test and declared the bug fixed. I asked whether it had actually tested it, and it hadn't. The test only passed because of a workaround for that very bug.
Trusting the wrong design. Tickets sometimes link the design for a different variant of a component. The agent compares against whatever it's given, so a wrong link means a confident report on the wrong thing.
Blaming the code for the environment. Test environments have their own problems: DNS that won't resolve, data that won't seed, a stale search index. Without context, the agent can spend a long time on something that isn't a bug.
So a human stays in the loop, and the guardrails are deliberate. It only tests non-production environments with test accounts. It never posts to Jira without my sign-off. Some actions, like reading environment secrets, are blocked outright by its permissions, and it asks me to do them.
What's left for the human
Quite a lot, as it happens. I run /test-ticket on every ticket I pick up, before doing anything else. But it's a first pass, not a sign-off. I spend less time clicking through forms at three screen widths and more time on what comes after it.
Checking its findings. I verify the problems it reports before raising them. A made-up bug wastes a developer's time and costs me credibility, so as a general rule nothing goes on a ticket unless I've seen it myself.
Finding the UI/UX problems it missed. Some of these it simply doesn't spot. Timing problems are the classic case: a flicker, a button that briefly shows the wrong state, a toast that disappears before the screenshot is taken. The agent sees the app as a series of snapshots, so anything odd that happens between two of them gets past it. Other problems come down to context. I've spent a long time using this product, so I know how a user's journey through it should feel. I can tell when a flow is awkward even though every AC passes, and the agent doesn't always have that context.
Spotting what it didn't test. The report lists what was covered, so a careful read shows the gaps. When something's missing, I either send the agent back to test it or test it myself, whichever is quicker. If the same gap keeps turning up, I fix the skill.
Trying it myself. I almost always still test the new feature with my own hands, mainly for the UX reasons above. The exception is a clear-cut backend change, where the diff and the report tell me everything I need.
The agent does the legwork, and I own the verdict.
If you want to try it, start small:
Get your tickets, PRs and designs linked, and get an environment per PR if you can.
Connect an agent to your ticketing system, your code and a browser.
Ask your AI agent to create a test-ticket skill file, and tell it everything you think a good QA tester should test for tickets in your project.
Run it on the next ticket, and edit it every time it disappoints you.
It might be a bit rough at first. Ours was. Within a few weeks of edits it was catching things I'd have missed.