How to test AI agents before go-live

Test an AI agent against a written definition of good, using real requests, and check every answer you can count with code instead of asking the model how it went. Then test the specific ways agents fail, keep a person on the risky steps, and re-run the whole set every time the model or the prompt changes.

This is for whoever has to say yes before an agent goes live. I have spent 18 years in software testing and test management, and the last few also building AI systems for real work at OIC Architects. The checklist below is what I run on my own agents. Most of it exists because something went wrong first.

What does good look like for this agent?

Write it down before anyone builds anything. One page is enough. It should say what the agent is for, what a correct result looks like, which failures are unacceptable, and which actions need a person's approval.

The National AI Centre's Guidance for AI Adoption, first published in October 2025, lists "test and monitor" as one of six practices. It asks for acceptance criteria that reflect the intended use and its risks, testing before deployment, and a documented go-ahead from the accountable person based on the results.

The definition also has to say what makes a score mean something. On an architect registration exam app I built, practice scores went up at one point. The test had got easier. The preparation had not improved. So the definition of good gained a rule that question difficulty must match the real exam. (The candidate later passed.)

How do you build an evaluation set for an AI agent?

An evaluation set is a list of inputs with the result you expect for each one. For the exam app, I calibrated the rubric against 37 sample questions from the exam board. For an agent, start with real requests from the people who will use it, then add the awkward ones:

requests with missing, irrelevant or empty input data
inputs far longer than normal
ambiguous requests where two people would disagree on the right answer
the same request phrased five or six ways, to see whether the agent picks the same tool each time
requests the agent should refuse or hand to a person

Run each case more than once, because the same agent, prompt and model can answer differently on each run. Count passes out of the number of runs, and set the pass rate before you look at the results.

Should the model grade its own work?

Not for anything you can count. On the exam app, an AI drafting pass reported that all 40 questions it had written met the rules. An independent check in plain code found 36 failures. I have followed one rule since. Never let a model self-report something you can count.

Check	Use it for	Weak spot
Code (deterministic)	Counts, formats, required fields, links, which tool was called	Cannot judge tone or reasoning
Model-graded	Judgement calls with a clear rubric, such as whether a summary kept every decision	Has to be tested against human grades before you trust it
Human review	High-stakes output, new kinds of failure, calibrating the other two	Slow, expensive and inconsistent when people are tired

Anthropic's evaluation guide agrees. Code-based grading is the fastest and most reliable but lacks nuance. Model-based grading is flexible and should be tested for reliability before you scale it. Hand the drafting to the agent, and keep the final checks with code and people.

Where do AI agents fail?

Agents fail in a few repeatable ways, and each needs its own test.

Failure	What it looks like	How I test for it
Made-up facts	A confident answer, citation or quote that does not exist	Check each claim against its source, with code where possible
Wrong tool call	The agent posts, writes or deletes in the wrong place	Many phrasings of one request, with a check on which tool fired
Prompt injection	Instructions hidden in a web page, email or file change what the agent does	Plant hostile text in test inputs and confirm the agent ignores it
Silent empty data	An input arrives empty and the output still looks finished	Assert that every input has content before the model sees it
Duplicates and overwrites	The same item processed twice, or good data replaced	Re-run on the same material and expect zero new items, and archive before any write

None of these is theoretical for me. In my content research system, Mandarin transcripts came back empty across 53 videos over 12 cycles, and nothing raised an error. In my meeting pipeline I found a duplicate transcript pair 4.5 seconds apart, and an agent once overwrote an action file with the text of its own permission request. That system now archives before it writes and validates what it wrote.

Made-up facts are the failure that makes the news. In October 2025 Deloitte agreed to repay part of its fee for a report to the Department of Employment and Workplace Relations. The report contained a fabricated quote attributed to a federal court judge and references to academic works that do not exist. Checking each reference against its source is dull work, and it would have caught both.

For prompt injection, start with the free OWASP Top 10 for LLM applications. It covers instructions hidden in content the model reads, and it recommends testing with the model treated as an untrusted user.

Where should a person stay in the loop?

At every step where a mistake is expensive or hard to undo. Drafting is fine to automate. Sending, deleting, paying and publishing get a person's click.

Two rules from my meeting pipeline have held up. Unknown names are flagged, never guessed. And completion is never inferred from silence. In the content research system, weak analysis produces a warning, but a fabricated claim about a real person stops the pipeline. That system needs about 20 minutes of human review a week, which is a fair price for trusting it.

What should you re-test when the model or the prompt changes?

Everything in the evaluation set. A new model version, a reworded prompt, a new tool or a new data source can each change behaviour you thought was settled. The guidance says it plainly. A system that worked well last month might give different answers today.

When three automated outputs in my meeting pipeline came back with inconsistent section structures, the fix went into the shared rules every output follows. Tuning the model would have hidden the problem until the next change.

Test your tests as well. While building this website with AI assistants, a text sweep reported zero copies of an old slogan, because the slogan was baked into an image. Now every new check gets broken on purpose first. If it does not fail when the thing it guards is broken, I do not trust it.

How do you monitor an AI agent once it is live?

Log every run, alert when an input source goes quiet, and sample real output against the rubric you used before launch. A green dashboard only proves the questions it asks. On the OIC Architects website, 1,351 links passed with none broken, yet the deploy folder held about 22 MB of staff photos that were about to go public. The build now copies only the files the pages use.

In the World Quality Report 2025, 60% of organisations named hallucination and reliability concerns as a barrier to using generative AI in quality engineering.

If you have an agent close to launch, write the one-page definition of good this week and build your evaluation set from it. If you want a second opinion on either, get in touch.

Questions people ask

How do you evaluate an AI agent's accuracy?

Run a fixed set of real requests through the agent several times and score each result against an expected answer. Use code for anything countable and a tested rubric for judgement calls. Report accuracy as a pass rate per type of request, because an overall average hides the case that fails every time. Agree the pass rate before you see the results, so nobody lowers the bar afterwards.

Can you unit test an AI agent?

Yes, for the parts that are ordinary code. Tool functions, input checks, output parsers and permission rules behave the same way every time, so normal unit tests work. The model's decisions need an evaluation set and pass rates instead. A good design keeps as much as possible deterministic, such as capturing data, formatting and routing, and gives the model only the judgement in the middle.

How do you stress test an AI agent?

Feed it the inputs that break things in practice. Extremely long documents, empty files, duplicates, contradictory instructions, another language and hostile instructions hidden in the content it reads. Watch for loops and runaway cost too. An agent that retries without a cap can keep going for hours, so set a retry limit and test that it stops.

How is testing an LLM application different from testing normal software?

Normal software gives the same output for the same input, so one expected result per test works. An LLM application can answer differently each time, so you test with repeated runs, pass rates and rubrics instead of exact matches. You still need requirements, test data, regression runs and a person accountable for the release. Most of the new work is in defining what a good answer is.

Sources

Share on LinkedIn

How to test an AI agent before it goes live (a test manager's checklist)