Stop Vibe-Checking Your AI Agent: Build an Eval Pipeline Instead

Building an AI agent is the easy part. The awkward question shows up right after you ship it: how do you actually know it works? And by that I mean: not on the three queries you tried by hand, but on the hundred you never got around to. This post is about the first evaluation pipeline I built for one of my agents, what it cost me in assumptions, and two results I got wrong. If you’re about to evaluate an AI feature for the first time, I hope it saves you a few of the detours I took.

When you ship an AI feature there’s a seductive little moment where it looks done. You type in a few questions, sensible answers come back, and your brain quietly files the thing under “works”. Then you change a prompt. Or swap a model. Or nudge a retrieval setting… and you realize you have no idea whether you just made it better or quietly broke it on the queries you never test.

“I poked at it and it looked fine” isn’t a strategy. What replaces it is evaluation, and building a real eval pipeline. This taught me more about the agent, and about my own assumptions, than coding it did.

One thing to set straight up front, because it colors everything below. This was a first iteration, not a final verdict on the agent. It’s the opening turn of a loop that’s supposed to keep running, and it raised more questions than it settled. For a first round, that’s the expected outcome.

A mini-glossary first

• Agent. A program that uses one or more large language models (LLMs) to make decisions: classify a request, decide what to do, call tools, produce an answer.
• Tracing / observability. Instrumentation that records what happened inside the agent: which steps ran, what the model saw, how long it took, what it returned.
• Evaluation (eval). A separate question: was what happened any good? Was the answer correct, relevant, faithful to the source?
• LLM-as-judge. Using a second, usually stronger, LLM to grade your agent’s output on things you can’t check with a simple rule.
• Faithfulness. Whether an answer sticks to the information it was given instead of confidently making things up. The polite word for making things up is “hallucination.”

The setup

The agent I was testing is small. It sorts a user’s request into one of a few intents, pulls up the relevant documents, and derives the proper context to answer from them. I’ll keep the domain vague on purpose; what matters is the shape. There’s a classifier, a retrieval step, a generation step, and a handful of knobs I could turn: which model to use, how much “reasoning effort” to allow, how strict to make the system prompt.

In other words, exactly the kind of system where “it looked fine” can hide a dozen quiet failures. What follows is what building the eval taught me, grouped into the handful of patterns I’d give anyone starting out.

Pattern 1: Tracing tells you what happened, evaluation tells you if it was good

This distinction matters more than it sounds. Tracing is your debugger. It shows you the agent’s internal monologue, the token counts, the timings. Evaluation is your test suite; it tells you whether the output was any good. You need both, and they answer completely different questions. I’ve seen teams with gorgeous tracing dashboards who still couldn’t tell you whether their agent actually worked, because tracing was never built to answer that.

Put another way: building an agent is maybe 30% prompt engineering and 70% figuring out whether the prompt engineering worked. Evaluation is that second 70%. Skip it and every change you make is just a guess with good lighting.

Pattern 2: Build the workflow first

I nearly walked straight into a trap here. Sitting down to design the first experiment, my brain went: “three models, times four prompt variants, times two retrieval strategies, times a hundred queries, times six metrics…”. That’s a 14,400-cell experiment. I was never going to run it, and neither are you.

The reframe that got me unstuck:

The goal of your first evaluation isn’t to find the optimal configuration. It’s to build the evaluation workflow.

Once the workflow exists (a dataset, a runner, some metrics, a way to compare two versions), adding a new variant is a one-line config change instead of a rewrite. So I cut the matrix down to five cells, each one isolating a single change from a baseline:

• A: the baseline, as a reference point.
• B: lower the model’s reasoning effort.
• C: a stricter system prompt.
• D: swap in a non-reasoning model (do we even need reasoning here?).
• E: the best two changes combined, as a candidate production config.

Everything else went into a deferred backlog: retrieval-k sweeps, hybrid retrieval, running each query more than once. I didn’t throw those ideas away, I just parked them. That backlog turned out to be useful in its own right, because the moment the first run flagged a real problem, my next experiment was already sitting there written down.

It’s an old software-engineering habit, really. Make it work first, then make it broad. Your first eval is an MVP, so resist the urge to polish it into something grand before it has ever produced a number.

Pattern 3: Not all metrics are equal, so pick your tiers

“How do I measure whether the agent is good?” is really three questions in disguise, and each one wants a different kind of measurement. I think about them in tiers, and I build them in this order.

Tier 1, deterministic. Free, exact, no model needed: latency, token counts, cost per query, classification accuracy wherever I have ground-truth labels. Start here. These gave me the most signal for the least effort, and the single biggest surprise of the whole project came from the cheapest metric on the list.

Tier 2, quasi-deterministic. An LLM-judge with a very tight rubric, for near-binary questions. Did the agent refuse when it should have? Did it follow the required format? You give the judge the criteria and ask for a 0 or a 1. Narrow enough that it stays consistent.

Tier 3, fuzzy LLM-judge. The genuinely subjective stuff: was the answer faithful, was it relevant, was it helpful. These cost real tokens, wobble from run to run, and need calibration, so I reach for them only when nothing cheaper will do.

You can usually answer 60 to 80% of “is this thing working?” with Tier 1 alone. The boring metrics are also where I learned the most. The token breakdown showed that some configurations were burning 85% of their output tokens on internal “reasoning” just to sort a query into one of three buckets. No fancy faithfulness score was ever going to catch a model thinking that hard about something that trivial.

Pattern 4: LLM-as-judge is powerful, but the judge needs grading too

Some questions can’t be checked with a regex. “Is this answer faithful to the context we retrieved?” is one of them. But you can ask another model. That’s all LLM-as-judge really is: you hand a second, usually stronger model the question, the context and the answer, and ask it to score whether every claim actually holds up.

You don’t have to write those judge prompts yourself. There’s an open-source library called RAGAS that ships the common ones for retrieval systems: faithfulness, answer relevancy, context precision and recall. I’d recommend reading its definitions even if you never run it. Seeing exactly how RAGAS computes “faithfulness” is the clearest explanation of the concept I’ve come across.

The catch, which I learned the hard way, is that the judge is also just a model, so it can be wrong. On one of my rubrics I got the polarity backwards for an edge case. It was scoring a “feature not built yet” stub as an attempted answer when it should have counted as a refusal. I only noticed because I sat down and hand-graded a few cases and thought, wait, that’s the wrong way round. So now I keep a short checklist.

• Use a judge that’s stronger than the models it’s grading.
• Lock it down, meaning the same model and the same prompt across every variant.
• Hand-grade five to ten examples before you trust a single score.
• Read the judge’s reasoning, not just the number it hands back.

Pattern 5: Be ready to be surprised

Then I actually ran it. I’d gone in with a few confident hypotheses. Two of them were flat wrong, and that turned out to be the most valuable thing the whole exercise produced.

Surprise one: “best plus best” came out worst. The obvious move was to take the two changes that each looked good on their own, combine them, and call that the production candidate. The combo did win on latency, and it won on cost. It also scored the worst on faithfulness of anything in the matrix. The two changes had interacted badly: a more instructable model plus a more structured prompt gave me a model that treated “use only the context” as a friendly suggestion and happily wandered off it. Quality dimensions don’t just add up. Best on X plus best on Y won’t hand you best on Z.

Surprise two: the “smart” model lost the easy job. I’d assumed the reasoning model would beat the cheaper non-reasoning one across the board, and especially at something as simple as sorting a query into three buckets. It didn’t. The cheaper model won on classification accuracy by a clear margin. With hindsight it makes sense. Short, well-bounded classification doesn’t gain anything from chain-of-thought, and the reasoning model was busy over-thinking a task that punishes over-thinking.

The lesson underneath both surprises is the same. You don’t run evals to confirm what you already believe. You run them to find the places where you’re wrong. If an eval agrees with every single one of your priors, be suspicious: either you got lucky, or your dataset is too narrow to turn up anything you didn’t already know. The real payoff isn’t “metric Z went up four points.” It’s the moment you catch yourself going “huh, I didn’t expect that, why?”

A caveat I want to be honest about: I ran each query once. At n=1 these are hypotheses, not conclusions. The right response to a surprising single-run result is to design a second experiment to confirm it, not to ship anything on Monday.

This was only round one

I want to be blunt about how little this first pass actually settled. Five cells out of an almost infinite space. Eighteen queries. One run each. That’s enough to build the machinery and shake loose a couple of sharp hypotheses. It is nowhere near enough to declare a winner and ship it.

That deferred backlog from earlier is really the roadmap for what comes next. The retrieval-k sweep, hybrid retrieval, running queries multiple times, a bigger and more adversarial dataset: those are experiments two, three and four, already lined up. Which is the point I most want to leave you with: an eval pipeline isn’t a project you finish and close out, it’s an instrument you keep using. Because I built the loop once, re-asking all of these questions after the next change to the agent costs me a config edit rather than another build from scratch. The first run is genuinely the least informative one I’ll ever do.

What I’d do differently

If I started over tomorrow, four things would change.

First, I’d plan for more than one run per query from the start. Single runs look tidy but they hide the noise, and a five-point gap can vanish on the second pass. The snag is cost: repeated runs multiply your token bill, and with pricier models that adds up fast. So n=1 is fine while you’re still standing up the infrastructure, but once you get to experiment two and want to trust a comparison, turn it up.

Second, I’d hand-label the retrieval ground truth early. An hour spent on “which documents were the right ones for this query?” unlocks a whole set of retrieval-quality metrics later.

Third, I’d build the report alongside the runner instead of bolting it on afterwards. Even a ten-line “print the summary” beats no summary at all.

Fourth, I’d calibrate the judges before the big run, not after the numbers start looking strange. See above, regarding the rubric I got backwards.

Final remarks

Almost none of this is new. Build the smallest thing that works before you generalize. Change one variable at a time. Don’t trust an instrument you haven’t calibrated. Write down the things you’re choosing not to do yet. These are old engineering principles in new clothes, and if anything the agentic era makes them matter more, because a model will happily produce confident, plausible, wrong output faster than any tool we’ve had before.

I started this project asking “I built an agent, is it any good?” I finished with a better question: “how do I know it’s getting better over time?” Moving from judging a single version to watching a trajectory is, for me, the real payoff of doing any of this.

If you’re looking at your own agent and wondering where to start, here’s the short version. Turn on tracing. Write fifteen or so labeled queries. Build a runner you can trigger with one command. Add the Tier 1 metrics. Compare two versions, and don’t believe anything you see on a single run. You can put that together in a couple of focused days, and everything fancier can wait until you’ve got it.

If you’ve been down this road already, I’d love to compare notes. I’m still chewing on a few things: how to think about statistical significance when every run costs real money, how to keep an eval dataset fresh without it turning into a chore, and how any of this changes once an agent has memory or runs over multiple turns.