How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python

                     How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python
               



Tópico:
                     How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python
               
Categoria: Tutoriais | FreeCodeCamp Premium
Idioma Principal: Português (Conteúdo de Tecnologia)

Conteúdo do Tutorial / Guia Passo a Passo:
-------------------------------------------------------------------------
In this tutorial, I'll show you how to evaluate a local AI agent with a simple, repeatable evaluation harness.

The harness runs the agent against a set of test cases, checks the results with both rule-based assertions and an LLM-as-a-judge, and prints a clear pass/fail summary.

Everything runs on your own machine with LangChain v1, Ollama, Qwen, and Python, so there are no API costs.

Table of Contents

• Background

• What is Agent Evaluation?

• What is LLM-as-a-Judge?

• Motivation and Architecture

• Step 1: Install Ollama and Pull the Model

• Step 2: Install Python Dependencies

• Step 3: The Agent Under Test

• Step 4: Write the Eval Harness

• Step 5: Run the Evals

• Sample Output

• Conclusion

Background

Most local AI agents get tested the same way: type a couple of questions, the answers look right, and just ship it. This works until we change the prompt, swap the model, or add a tool. Then something breaks quietly, and we don't notice until it's too late.

Regular Python code has unit tests to catch this. AI agents don't get that for free. Even with the same input, an agent can behave differently across runs, and small changes can introduce regressions that are easy to miss. Without a repeatable way to test the agent on multiple inputs and score the outputs, we're mostly guessing on agent's behavior.

A simple fix is to build a lightweight evaluation setup that contains a Python script, a list of test cases, rule-based checks, and an LLM-as-judge. That gives us a practical way to test the agent before on any changes.

To follow along, you'll need Ollama installed on your machine. The tutorial works on macOS, Windows, and Linux. I'm using a MacBook Pro with 32 GB of RAM, but you can run this on a lower-memory machine by choosing a smaller Qwen model from Ollama.

What is Agent Evaluation?

Agent evaluation is the practice of running your agent against a fixed set of inputs and scoring the outputs against expectations. It's the AI equivalent of a test suite.

The goal isn't to prove the agent is perfect. The goal is to catch regressions when you change something.

A useful eval has three parts:

• Test cases: a list of inputs with expected behaviors.

• Checks: functions that score the agent's output for each input.

• A summary: a pass/fail count so you can see how the agent did.

What is LLM-as-a-Judge?

There are two practical ways to score an agent's output. The first is rule-based checks. You assert on things like "did the output contain the word Paris" or "did the agent call the

Código Selecionar Expandir

word_counttool." These are cheap, fast, and deterministic.

The second is LLM-as-a-judge. You ask a separate LLM to read the input and the agent's output, then score it against a rubric. A rubric can be a simple pass/fail output. This is useful for fuzzy things you can't easily assert on, like "did the answer actually address what the user asked." The tradeoff is that the judge is itself an LLM and can be wrong.

In this tutorial, we'll be using the same model with a different prompt for judging.

Motivation and Architecture

Evaluating an agent is the natural next step after building one. Knowing the agent works reliably across different inputs is what turns it into something we can trust.

To keep things simple, we'll evaluate a small local agent with two tools: one for the current time and another for c

... [O tutorial continua no link abaixo] ...