We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.



Tópico: We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one question a merge queue cares about: does this gate pass or fail the same way twice. I wired six of these frameworks into a real GitHub Actions merge queue and ran them against production PRs for about eight months. The ones that gate cleanly share a single property: deterministic checks that return an exit code in seconds, with LLM-as-judge scores kept as non-blocking signals. The ones that flake share the opposite: nearly every metric is a judge call, so the queue blocks on a number that drifts. Ranked by "survived our merge queue," Promptfoo and DeepEval came out ahead. The short list first, then per-tool notes, then when you should not gate on any of them.



The outage that set the ranking


Two years ago I put an LLM-as-judge metric on our merge queue with a 0.8 threshold. It looked clean in the demo. Three weeks later it blocked fourteen PRs and a release over a weekend, because the judge scored the same unchanged output 0.83 on Friday and 0.78 on Monday. Same prompt. Same model. No seed. I killed the gate at 1am from my phone and we shipped fine. The regression it was "protecting" us from never existed.

That is the lens for this whole piece. A CI gate has one job: fail when something broke, pass when it did not, and do it the same way every time. An eval framework can carry the best metrics in the world and still be a bad gate if those metrics wobble. The feature-ranked listicles miss this because a notebook never punishes you for nondeterminism. A merge queue does, at 1am, in front of the whole team.

I am not arguing that quality evals are useless. I run plenty of them. I am arguing that a merge queue is a specific, unforgiving place, and the tool that belongs there is not always the tool with the longest metric list. A gate that blocks a clean PR trains your team to force-merge past it, and once the team routinely force-merges past a gate, it has stopped protecting anything. It only adds a click everyone has learned to ignore. That is the failure mode I now rank against first.



What "survived the merge queue" means


Five things, in the order they bit me:

• Determinism. Same input, same verdict. A judge call with no fixed seed is not deterministic, and no threshold tuning will fully save you.

• Speed. If one eval run adds four minutes and you merge forty PRs a day, you have bought a queue backup.

• Cost. Judge-graded metrics burn tokens per run. Multiply by PR count. Some months that is a real invoice nobody budgeted for.

• Wiring effort. Does the tool return an exit code, or do I hand-build the pass/fail logic around it.

• Signal quality. When it fails, does it point at the actual regression, or just report a lower number and shrug.

Everything below is graded on that, not on how good the metrics look in a demo. Two of those five are about determinism and its side effects, because that is what cost me the most sleep. The other three (speed, wiring, signal) are what decide whether the gate is worth keeping once it works.

How I ran it: each tool guarded the same small golden set (about 60 input/expected pairs for a support-answer feature) inside the same GitHub Actions workflow, blocking on merge. I rotated them one at a time and watched three numbers: the flake rate on unchanged inputs, the added minutes per run, and the token bill at the end of the month. I kept a metric in the blocking path only if it never flipped a verdict on an input that had not changed. That rule alone reshuffled the list.

All versions, licenses, and metric counts are as of mid-2026. Check each repo before you rely on any of it, because all of this moves.



At a glance


• 1. Promptfoo (MIT). CI hook: a CLI exit code, JSON output, and a maintained GitHub Action. Metrics: dozens of assertions, deterministic and graded. LLM-judge: optional. Best for CLI gating in any stack.

• 2. DeepEval (Apache-2.0). CI hook: a pytest wrapper (deepeval test run). Metrics: 20-plus. LLM-judge: default for most. Best for pytest-based Python gates.

• 3. Future AGI (Apache-2.0). CI hook: call evaluate() in your own harness. Metrics: 50-plus (local plus hybrid judge). LLM-judge: optional. Best for an eval SDK you drive from Python or TypeScript.

• 4. RAGAS (Apache-2.0). CI hook: call evaluate() in a script. Metrics: around a dozen RAG-specific. LLM-judge: yes, most. Best for RAG quality measurement.

• 5. Arize Phoenix (Elastic License 2.0). CI hook: run_evals() in a script. Metrics: a handful of evaluators. LLM-judge: yes. Best for tracing plus eval in one tool.

• 6. MLflow evaluate (Apache-2.0). CI hook: mlflow.evaluate() in a script. Metrics: a dozen-plus (heuristic plus genai). LLM-judge: optional. Best for eval logged alongside experiments.

Repo links and install lines are in each section. Now the details.



The pattern that decided it


Once all six were in the same workflow, the ranking stopped being about metric quality. It came down to one split: does the blocking check call a model, or not. Deterministic checks (string match, JSON-schema, regex, exact match) passed and failed the same way every run, finished in under a second, and cost nothing. Judge checks did not. Every judge-based gate I ran drifted near its threshold at least once over the eight months, and two of them blocked a clean PR at least once. So the tools that let me put deterministic assertions in the blocking path, and push judge scores into an advisory lane, came out ahead. The tools built around a judge-first metric set fell behind, not because the metrics are weak, but because a score that moves between identical runs cannot hold a merge gate. Keep that split in mind as you read the six. It explains the whole order, including why a well-known RAG library sits below a younger tool, and why the tracking and observability tools sit at the bottom even though their metrics are fine.



1. Promptfoo


Repo: github.com/promptfoo/promptfoo. License: MIT. Install: npm install -g promptfoo (or run it with npx promptfoo).

What it is. A command-line eval and red-teaming tool. You describe prompts, providers, and test cases in a YAML file, run one command, and get a pass/fail table. As of mid-2026 it ships dozens of built-in assertions split into two camps: deterministic ones (contains, equals, regex, is-json, starts-with, cost, latency) and model-graded ones (llm-rubric, factuality, answer-relevance, similarity by embedding).

How it gates CI. promptfoo eval returns a nonzero exit code the moment an assertion fails. That is the whole ballgame for a merge queue: a nonzero exit is a red check, no glue code required. It writes JSON, CSV, or HTML you can archive as a build artifact, and there is a maintained GitHub Action that comments the eval diff on the PR so a reviewer sees exactly what changed. In my run the deterministic assertions never flaked across the eight months. The only week the queue wobbled was one where I let an llm-rubric assertion sit in the blocking path, and the fix was to move it back out.

# promptfooconfig.yaml  ->  run in CI with: npx promptfoo eval -c promptfooconfig.yaml
prompts:
- "Answer the support question: {{question}}"
providers:
- openai:gpt-4o-mini
tests:
- vars:
question: "What is the refund window?"
assert:
- type: contains
value: "30 days"        # deterministic: no judge, no flake, sub-second
- type: llm-rubric
value: "states a clear refund window"   # graded: costs a call, can flake

Strengths. The deterministic assertions are the reason it sits at the top. contains and is-json do not flake, cost nothing, and run in milliseconds. It is language-agnostic: a Python shop, a Go shop, and a TypeScript shop all wire it the same way, because it is a CLI that speaks exit codes. Config lives in version control next to the code it guards, so a bad gate change shows up in the same diff.

Limits. The YAML sprawls once you pass a few dozen cases, and there is no type safety net until you run it. The model-graded assertions carry the same judge nondeterminism as everything else here, so if you lean on llm-rubric to block merges you have reintroduced the flake you came to avoid. A Python-first team also takes on a Node dependency it may not otherwise want in the CI image.

Best for. Gating prompt and output regressions in any stack, as long as you keep the blocking assertions deterministic and treat the graded ones as advisory.



2. DeepEval


Repo: github.com/confident-ai/deepeval. License: Apache-2.0. Install: pip install deepeval.

What it is. A Python eval framework built to feel like pytest. As of mid-2026 it carries 20-plus metrics, including G-Eval (a rubric metric you define in plain language), answer relevancy, faithfulness, hallucination, contextual precision and recall, plus safety metrics like bias and toxicity. Most of them are LLM-judged.

How it gates CI. You write a normal-looking test file and run deepeval test run test_file.py. Under the hood it wraps pytest, so you inherit pytest's exit codes and its JUnit XML reporter for free. If your CI already understands pytest, it already understands DeepEval. That is the shortest path to a green check for a Python team in this whole list. In my run the pytest wiring took under an hour to stand up. The flake, when it came, came entirely from the judge-based metrics drifting near their thresholds, not from the harness.

# test_support_bot.py  ->  run with: deepeval test run test_support_bot.py
import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

# The judge is an LLM, so the score is not deterministic.
# A 0.7 gate that scores 0.71 today can score 0.68 tomorrow on the same input.
CASES = [
("What is the refund window?", "You can request a refund within 30 days."),
("What is the refund window?", ""),   # empty output: the edge case that bites
]

@pytest.mark.parametrize("query,answer", CASES)
def test_answer_relevancy(query, answer):
metric = AnswerRelevancyMetric(threshold=0.7)
case = LLMTestCase(input=query, actual_output=answer)
# Empty actual_output does not raise; it scores near 0 and fails the gate,
# which is correct. The flaky judge near the threshold is the real risk.
assert_test(case, [metric])

Strengths. The pytest wrapping is the cleanest on-ramp here for a Python team. Per-metric thresholds are explicit and live in code. The catalog is the widest in this list, so you rarely hand-roll a metric, and G-Eval lets you encode a house rule ("must cite a ticket id") without writing a scorer from scratch.

Limits. Most headline metrics are judge calls, so the flakiness story from my outage applies directly: keep thresholds loose or your queue pays for it. It nudges you toward Confident AI, the hosted product from the same team, once you want dashboards and shared datasets. And a wide catalog is a wide surface to keep pinned, because a judge-model upgrade can shift scores under a fixed threshold with no code change on your side.

Best for. Python teams that already gate on pytest and want deterministic wiring, as long as you pick a few metrics and treat the judge-based ones with suspicion near the threshold.



3. Future AGI


Repo: github.com/future-agi/future-agi. License: Apache-2.0. Install: pip install ai-evaluation.

What it is. An open-source eval SDK (fi.evals). As of mid-2026 its README lists 50-plus evaluation metrics plus guardrail scanners, behind one evaluate() call, with Python and TypeScript clients. It is one piece of a larger open-source platform, but for CI gating only the eval SDK matters, so that is all I put in front of the merge queue.

How it gates CI. There is no dedicated test runner. You construct an Evaluator, call evaluate() over your inputs, and write your own assert on the returned scores, the same shape as RAGAS below. One wrinkle worth knowing for a gate: the documented quickstart authenticates with an API key and runs against the hosted service, so the naive setup puts a network call in your blocking path. The SDK also supports local metric execution, which is the mode you want for a merge gate, because a deterministic local metric does not flake the way a judge does and does not add a per-run token bill. Turn the hybrid judge on and you inherit the same nondeterminism as everyone else in this list. The actual time sink was the harness code I had to write around evaluate() myself, because nothing here hands you a runner.

# pip install ai-evaluation   (module: fi.evals)
from fi.evals import Evaluator
# documented quickstart authenticates with API keys (hosted execution)
evaluator = Evaluator(fi_api_key="...", fi_secret_key="...")
result = evaluator.evaluate(eval_templates="...", inputs={...})   # signature abbreviated
# no test runner: read the score off `result` and assert it in your own gate

Strengths. When you run its metrics in local mode, a deterministic score is what a gate wants: same input, same score, no per-PR token bill. One evaluate() surface for both local and judge scoring keeps the harness small. The TypeScript client means a Node CI job can gate without shelling out to Python, which is rare in this list.

Limits. It is younger than DeepEval and Promptfoo and it shows. There is no first-class pytest plugin or JUnit reporter, so you write more of the harness yourself. The documented quickstart is hosted (API key), so you have to configure local execution yourself to keep a network call out of the blocking path. The community, examples, and CI recipes are thinner, which matters at 2am when the gate breaks and you are hunting for the one forum answer that does not exist yet. On raw metric breadth and CI-native ergonomics it does not beat DeepEval, and it is not the most mature option on this list. If you want a tool that gates straight out of the box, this is not the shortest path today.

Best for. Teams that already have a CI harness and want fast local metrics to call from it, in Python or TypeScript, without paying a judge-token bill on every PR.



4. RAGAS


Repo: github.com/explodinggradients/ragas. License: Apache-2.0. Install: pip install ragas.

What it is. A library focused on retrieval-augmented generation. As of mid-2026 it offers around a dozen RAG-specific metrics: faithfulness, answer relevancy, context precision, context recall, answer correctness, and a few newer ones. The metrics are well-researched and map cleanly onto the stages of a RAG pipeline.

How it gates CI. There is no runner. You build a dataset, call evaluate(), and get back a scores object. Turning that into a gate is on you: read the metric, compare to a threshold, raise.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
# dataset is a HF Dataset with question / answer / contexts columns
scores = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
assert scores["faithfulness"] >= 0.9   # you write the gate yourself

In my run the scores moved enough between two identical runs that I could not keep it in the blocking path, so it lived in a nightly job instead. Why it sits below a younger tool here: nearly every RAGAS metric is a judge call. This ranking is about the merge queue, and judge calls are the exact thing the merge queue punishes. This is not a knock on RAGAS as software. For the narrow job of a hard merge gate, the judge dependency is what drops it below a younger tool.

Strengths. If your problem is specifically RAG quality, these are among the most thought-out metrics available, and the decomposition (retrieval versus generation) tells you where the regression lives, not just that one happened. For diagnosing a bad retrieval step, that split is worth a lot.

Limits. RAG-only by design. Judge-based, so nondeterministic and token-costly per run. Scores drift when the judge model or its version changes, which turns a green history red with no code change. You own all of the pass/fail plumbing.

Best for. Measuring RAG retrieval and generation quality, ideally in a nightly or pre-merge advisory job rather than a hard blocking gate.



5. Arize Phoenix


Repo: github.com/Arize-ai/phoenix. License: Elastic License 2.0 as of mid-2026 (source-available, not OSI-approved; confirm in the repo, because this kind of license has changed before). Install: pip install arize-phoenix.

What it is. Primarily an observability tool. It ingests OpenTelemetry traces of your LLM app and gives you a local UI to inspect them. It also ships phoenix.evals, a library with a handful of prebuilt LLM evaluators (hallucination, QA correctness, relevance, toxicity) and a run_evals harness.

How it gates CI. You can call run_evals in a script, get a dataframe of labels or scores back, and assert on the aggregate.

from phoenix.evals import run_evals, HallucinationEvaluator, OpenAIModel
# returns a dataframe of labels/scores; assert on the aggregate  (API abbreviated)
evals_df = run_evals(dataframe=df, evaluators=[HallucinationEvaluator(OpenAIModel())])
assert (evals_df["label"] == "factual").mean() >= 0.95

It works, but gating is not the design center. The product is built around the trace UI, where you sit and inspect runs. A blocking merge gate is not what it optimizes for. In my run I got more out of pointing it at captured traces than at a merge gate. As a blocker it was awkward, and I kept reaching for the UI to understand a failure instead of reading an exit code.

Strengths. If you also want tracing, this is the one tool here that does eval and observability under a single install, so your CI check and your production debugging speak the same vocabulary. The classification-style evaluators (a label, not a free-form score) are more gate-shaped than raw judge numbers.

Limits. The license is the first thing to run past legal, because source-available is not the same as open source and some orgs treat that line as a hard stop. Gating is a bolt-on, so you build the pass/fail yourself. The evaluators still lean on a judge, with the usual nondeterminism and token cost.

Best for. Teams that want tracing and eval together and will run evals mostly over captured traces, with CI gating as a secondary use.



6. MLflow LLM evaluate


Repo: github.com/mlflow/mlflow. License: Apache-2.0. Install: pip install mlflow.

What it is. mlflow.evaluate() is the LLM-eval entry point inside MLflow, the experiment-tracking platform. As of mid-2026 it offers a dozen-plus built-in metrics, split between heuristic ones (toxicity, reading-grade, exact match, ROUGE, token count, latency) and genai ones you build with make_genai_metric that call a judge.

How it gates CI. You call mlflow.evaluate() on a model or a static dataset, it returns a results object, and you read a metric off it and assert. The heuristic metrics are deterministic, which is the good news for gating. In my run the exact-match and ROUGE metrics held steady as a gate. The genai metrics behaved like every other judge here, and the run-and-experiment ceremony was more setup than a single CI check wanted.

import mlflow
# heuristic metrics are deterministic; genai metrics call a judge
results = mlflow.evaluate(data=eval_df, model_type="question-answering")
assert results.metrics["exact_match"] >= 0.9   # key names vary by version

Strengths. If you already live in MLflow, your eval numbers land next to your training runs and artifacts, with lineage, which is genuinely useful for audits and post-incident review. The heuristic metrics do not flake, so a gate built on exact match or ROUGE holds steady.

Limits. It is built for experiment tracking, not gating, so reducing a run to one clean pass/fail feels like fighting the grain. It expects a run and an experiment context, which is a lot of ceremony for a CI check. The genai metrics reintroduce judge nondeterminism, and MLflow is a heavier dependency to pull into a lean CI image than a single-purpose eval library.

Best for. Teams already standardized on MLflow that want eval logged alongside experiments, using the heuristic metrics for any actual blocking gate.



When not to gate CI on any of these


Picking the right tool does not mean you should gate at all. The best framework in the world is the wrong call in some situations, and I have watched teams (mine included) reach for a merge gate when the real problem was upstream. Gating is not free, and sometimes it is the wrong move no matter which tool you pick:

• You have no golden dataset. If you cannot say what the right answer looks like, an eval gate just encodes a guess and fails at random. Build the dataset first, gate second.

• Your output is open-ended. Marketing copy, brainstorms, and open chat have no single correct answer for a judge to hit. Gate the structure (valid JSON, required fields present), not the quality.

• The judge bill beats the value. Forty PRs a day times a multi-metric judge run is a line item. If nobody will defend that spend, move the eval to nightly.

• No one owns the drift. Judge scores move when models update. If no human owns re-baselining, the gate rots into a check everyone force-merges past, which is worse than no gate at all.

• Latency breaks your SLA. If the eval adds minutes and your team merges constantly, you have traded correctness theater for a queue backup.

In those cases the answer is usually an async nightly eval, an online eval on a canary, or a deterministic structural check in CI with the quality eval running out of band where it cannot page anyone.



What I'd check first


• Run the gate 20 times on one unchanged input. If the pass/fail flips even once, treat it as an advisory signal and make it non-blocking.

• Time a single run and multiply by your daily PR count. If the minutes or the judge-token bill blow your queue budget, move it to nightly before you tune a single threshold.

• Delete your flakiest metric and see whether one deterministic assertion (regex, JSON-schema, contains) catches the same regression. It usually does, and it never pages you at 1am.