# How Do You Test What an AI Agent Says? Author: Cory McAboy Published: 2026-11-12 Source: https://rapptr.com/blog/ai-evals-testing-ai-agents/ Categories: Artificial Intelligence, AI I got a good question this week. How do you test the outputs of a chatbot or a voice agent, and how do you do it at scale? It came from a team running a voice agent that talks to real people every day. They want to know that what the agent says is accurate and on script, and nobody has time to listen to every call. The answer is something the AI world calls evals. [Embedded media]() ## Why normal QA does not cover this Traditional testing works because software is predictable. You click a button, you expect a specific result, and the test passes or fails. An AI agent does not behave that way. Ask it the same question twice and you will get two different answers. Both might be fine, or one might quote the wrong price. A test that checks for an exact match cannot tell the difference. ![One customer question with three AI agent answers. Two are worded differently but correct, and one quotes the wrong price.]() You can have people read transcripts and listen to calls, and you should do some of that. It stops working once the agent is handling hundreds of conversations a day. ## What an eval is An eval is a repeatable test for AI behavior. You start with a set of sample inputs, like real questions customers have asked. For each one, you define what a good answer looks like. Then you run the agent against the set and score every response. ![A four-step eval flow. Sample inputs go to the AI agent, its responses are scored by code checks, a model grader and a human spot check, and the result is 8 of 9 passed.]() The scoring can happen in a couple of ways. Some checks are plain code, such as whether the answer included the required disclosure or stayed under a certain length. Other checks need judgment, so a second model grades the answer against a rubric that you wrote. A person still reviews a sample of those grades to make sure the grader can be trusted. At the end, you have a score instead of a gut feeling. ## Why evals matter more over time Agent instructions change constantly. Someone edits the prompt to fix one complaint, or the team moves to a newer model. Each change can quietly break an answer that used to be right. Companies still want the same consistency in the answers. Evals give you that. You run the same set after every change and see what got better and what got worse before your customers do. ![The same nine questions run against two prompt versions. Version one passes 9 of 9, and version two passes 7 of 9.]() If you have a regression suite for your app, this is the same idea applied to what the AI says. ## What it takes to set up Evals are their own discipline, with their own tools and practices. Observability platforms like LangSmith and Langfuse record what your agent is doing in production and let you run evals against it. The type of AI application matters too. If your agent answers from your own documents, which is called RAG, you also need to test whether it pulled the right documents in the first place. There are separate tools for that, like Ragas. The biggest investment is usually the data. Someone on your team has to supply real conversations and say which answers were good and which were not. Without that, there is nothing to test against. ## How this scales People ask what happens when the agent is handling thousands of conversations. There are two kinds of evals, and you need both. The first kind runs before you ship. You keep a fixed set of test cases and run it every time someone changes the prompt or the model, the same way your regression tests run before a release. The second kind runs on live conversations. There is no answer key for a real call, so the grader looks for problems instead, like a missing disclosure or a promise the agent should not have made. You do not have to grade every conversation. Grading a sample keeps the cost down and still shows you when quality slips. The two feed each other. When the live checks flag a bad answer, you add that conversation to your test set, so the same mistake gets checked on every future run. This is where platforms like LangSmith and Langfuse earn their keep. They store the conversations and your test sets, and they run the graders automatically, so you are not building that plumbing yourself. They do not decide what a good answer is. Your team still has to write that down. ![The LangSmith experiment view, a table of test inputs, reference outputs and agent outputs with a score for each evaluator.]() _An eval run in LangSmith, with a score for every test case. Image from the [LangSmith documentation]()._ ## Where to start You do not need to cover everything on day one. Start with the short list of things your agent can never get wrong, like pricing or what it promises a customer. Collect a few dozen real conversations that touch those topics and write down what a correct answer looks like for each. That small set will catch more problems than you expect, and you can grow it from there. ## We can help At Rapptr Labs, we build AI features into the products we ship, and our QA team already [uses AI agents in its testing work](). If you are trying to figure out how to test what your chatbot or voice agent is saying, reach out. I'm happy to talk it through. [**Get in touch with Rapptr**]() ## Frequently Asked Questions **What are AI evals?** AI evals are repeatable tests that score the outputs of an AI system, such as a chatbot or voice agent. You run a fixed set of sample inputs through the system and grade each response against a definition of a good answer. **How do you test a chatbot or voice agent at scale?** You build an eval set from real conversations and score the agent's responses automatically. Simple rules are checked with code, and answers that need judgment are graded by a second model against a rubric, with people reviewing a sample of the grades. **Why can't traditional QA test AI outputs?** Traditional tests expect one exact result. An AI agent can give a different answer each time, and more than one answer can be correct, so exact-match tests do not work. **What tools are used for AI evals?** Observability platforms such as LangSmith and Langfuse support evals. Applications that answer from a company's own documents often add a tool like Ragas to test the retrieval step. **What do you need before you can run evals?** You need real examples of the conversations your agent handles and a clear definition of a good answer for each one. Gathering and labeling that data is usually the largest part of the work.