AI Evals Architecture · June 2026

How AI Evals Actually Work

A technical research page on how modern AI evaluation systems execute tests, capture traces, score RAG and agent behavior, and turn quality checks into CI gates and production monitoring.

Date1 Jun 2026

AudienceFounder-engineers

MethodShared transcript + primary-source research

Length~5,500 words

The unit of evaluation has moved from answer to trace. Simple prompt tests judge a final string. Serious RAG and agent systems need the retrieval span, tool call, judge prompt, score, reason, dataset version, model version, latency, cost, and failure location.¹²³
RAG evals are diagnosis, not a single score. A useful RAG eval separates context precision, context recall, faithfulness, answer relevance, citation support, and retrieval latency so a team can tell whether parsing, retrieval, prompting, or generation failed.⁴⁵⁶
Agent evals are trajectory evals. For agents, the important object is the path: messages, tool names, arguments, observations, order, retries, refusals, and final synthesis. The same final answer can hide unsafe or wasteful intermediate behavior.²⁷⁸
LLM judges are useful but must be treated as systems. A judge score is only meaningful when the judge prompt, model, schema, examples, reasoning, calibration set, and human agreement are logged and versioned.⁹¹⁰¹¹

3layers

Execution harness, evaluator layer, and observability/control plane

2modes

Offline regression gates plus online production scoring

6judge types

Deterministic, reference, LLM, pairwise, human-aligned, and security scorers

13tools

Platforms and libraries mapped by architectural role

Inference

Read this as an architecture map: the important question is not which eval tool has the longest feature list. The important question is where the tool sits in the system: runner, metric library, trace store, judge layer, gateway, CI gate, or production monitor.

01The failure that starts the story

Imagine a support RAG system gives a polished answer about refunds. The answer is concise, friendly, and grounded in the chunks that were retrieved. A final-answer judge says it is relevant. The team ships the prompt change.

Then a customer escalates. The answer was wrong because the exception lived in a table that was broken during ingestion, missed by retrieval, and never shown to the generator. The failure was not visible in the final answer. It was visible only if the eval system had captured the trace.

01
Question
looks fine
User asks: Can I get a refund after 45 days if the item failed?
The user intent depends on product category, warranty language, and an exception buried in a policy PDF.
02
Ingestion
silent risk
The source PDF was converted to markdown and embedded last week.
The table containing the warranty exception was split across chunks, so the relevant row never became searchable evidence.
03
Retriever
partial pass
Top-k includes the general 30-day refund policy.
The retriever missed the exception clause. A final-answer judge may call the response concise while context recall is actually failing.
04
Generator
confident wrong
The model answers that the refund window is 30 days.
The answer is faithful to the retrieved context but not faithful to the source corpus. That is why faithfulness alone is not enough.
05
Judge
needs audit
The LLM judge gives a high relevance score.
The judge prompt did not receive reference contexts, so it could only evaluate answer style and local support, not source completeness.
06
Decision
business impact
The eval run passes and the prompt ships.
A production customer later gets the wrong refund decision. The missing signal was in the trace, not in the final output string.

Caution

This is the core reason modern evals are moving beyond answer grading. A final string can look correct while retrieval, parsing, tool use, policy, or judge configuration is broken.

02The common eval machine

Most eval products look different at the UI layer, but the machine underneath is surprisingly stable. They start with examples, run the system under test, capture what happened, score the behavior, and persist enough evidence for comparison.

LangSmith expresses this through datasets, experiments, evaluators, and traces. DeepEval turns test cases and metric objects into thresholded checks. Ragas focuses on the metric call over normalized RAG samples. Promptfoo makes the runner declarative. OpenAI Evals gives API objects for evals, runs, graders, and data sources.1 4 5 12 14

Dataset

Defines the examples

Inputs, expected outputs, references, rubrics, personas, gold documents, allowed tools, adversarial prompts, and metadata. This is the test fixture for probabilistic software.

Runner

Executes the target

Calls a prompt, model, RAG pipeline, API endpoint, or agent graph across every example. Runners may execute locally, in CI, in a vendor platform, or against production samples.

App Under Test

Produces behavior

The actual system: a prompt template, retrieval chain, tool-calling loop, multi-agent workflow, model config, or product endpoint.

Trace Capture

Makes behavior inspectable

Records spans for model calls, retrievers, tools, parsers, rerankers, guardrails, user feedback, tokens, cost, errors, timing, and parent-child relationships.

Scorer Layer

Turns behavior into signals

Runs deterministic assertions, reference checks, semantic similarity, LLM judges, pairwise comparisons, task-completion checks, policy checks, and security probes.

Score Store

Preserves the evidence

Persists scores, reasons, judge prompts, model versions, dataset versions, trace links, run metadata, and experiment comparisons.

Gate Or Monitor

Changes engineering behavior

Blocks a pull request, flags a regression, alerts on production drift, queues human review, or tells the team which span failed.

03The record shape

Evals become portable when the system can emit the right record. A RAG eval often needs only the question, answer, retrieved contexts, reference answer, and reference contexts. An agent eval needs the same outer shell plus the sequence of messages, tool calls, tool arguments, observations, and final state.

That record shape is why metric libraries can sit inside full platforms. Ragas can compute metrics over a dataframe or dataset. DeepEval can score an `LLMTestCase`. LangSmith, Phoenix, Langfuse, and Braintrust can attach scores back to traces and experiments.2 4 5 11 16 17

RAG evaluation record

The portable tuple most RAG metrics need

{  "question": "What is the refund exception for failed items?",  "answer": "Refunds are available for 30 days.",  "retrieved_contexts": ["General policy: 30 days", "..."],  "reference_contexts": ["Failed items retain warranty coverage"],  "ground_truth": "Failed items can be refunded after 45 days if warranty applies.",  "metadata": {    "retriever": "hybrid-bm25-vector",    "top_k": 8,    "corpus_version": "policies-2026-05-30"  }}

Agent trajectory record

The behavior path, not only the final answer

{  "input": "Find the invoice and email the customer a summary.",  "final_output": "I found invoice INV-1042 and drafted the email.",  "trajectory": [    { "type": "tool_call", "tool": "search_crm", "args": { "customer": "Ava" } },    { "type": "tool_result", "output": "Two matching customers" },    { "type": "tool_call", "tool": "send_email", "args": { "recipient": "first match" } }  ],  "expected_behavior": "Disambiguate customer before sending.",  "policy": "No external email without explicit confirmation."}

Evidence

The best early design decision is not choosing a vendor. It is making every AI run emit a normalized trace and a normalized eval record. Once that exists, metric libraries, hosted dashboards, local CI, and human review workflows can all plug in.

04RAG evals

RAG evals are easiest to understand as fault isolation. Retrieval metrics ask whether the right evidence arrived. Generation metrics ask whether the answer uses that evidence correctly. End-to-end metrics ask whether the user got a useful, correct, and supported answer at acceptable latency and cost.

Ragas and DeepEval both expose this layered model through metrics such as context precision, context recall, answer relevance, and faithfulness. Phoenix and other observability-first tools add the trace view that explains which span created the bad answer.4 5 6 16

Context precision

Retriever ranking

Are useful chunks ranked above distracting chunks?

Compare retrieved chunks with gold evidence, or ask a judge whether each chunk supports the query.

Context recall

Retriever coverage

Did the retriever fetch all evidence needed to answer?

Compare retrieved contexts with reference contexts or known supporting documents.

Context relevance

Retrieved context

Is each chunk actually relevant to the user question?

Use embeddings, cross-encoders, or LLM judges to score chunk-query relevance.

Faithfulness

Generator grounding

Are the answer claims supported by the retrieved context?

Decompose answer claims and verify each claim against context using a judge or entailment model.

Answer relevance

Final answer

Does the answer address what the user asked?

Use reference-free judge prompts, semantic similarity, or generated-question similarity.

Citation support

Evidence links

Do cited sources support the exact claims they are attached to?

Map claim spans to source spans and judge whether each citation is sufficient.

Latency

Runtime health

Which step makes the answer too slow?

Read timing from trace spans: parse, retrieve, rerank, generate, judge, and post-process.

Cost

Economic health

Is quality improving at an acceptable token and judge-call cost?

Aggregate prompt tokens, completion tokens, embedding calls, reranker calls, and judge calls per run.

05Agent evals

Agent evals are harder because the product is not just an answer. It is a sequence of choices. A customer support agent can end with the right summary while querying the wrong customer first. A booking agent can find a good flight while skipping confirmation. A coding agent can make tests pass while editing files it should not touch.

This is why agent evals center on trajectories: tool names, tool arguments, observations, message order, retries, refusals, and handoff points. AgentEvals focuses directly on this behavior, while trace-first platforms can score the same path from recorded spans.2 7 8

Task success

Outcome

Did the agent actually complete the user goal?

Score final state against a rubric, reference outcome, environment state, or human label.

Tool selection

Tool calls

Did the agent choose the right tools for the task?

Compare actual tool names against expected, allowed, or forbidden tool sets.

Argument correctness

Tool inputs

Were tool parameters complete, valid, and safe?

Validate schemas and compare arguments with expected values, policies, or environment facts.

Tool order

Trajectory

Did the agent take actions in a safe and sensible sequence?

Score message and tool-call order with deterministic rules or a trajectory judge.

Policy compliance

Safety boundary

Did the agent avoid forbidden actions and permission violations?

Check trace spans for policy breaches, sensitive tools, data exfiltration, or missing approvals.

Efficiency

Step budget

Did the agent solve the task without unnecessary loops or calls?

Track step count, repeated calls, latency, tokens, retries, and cost against thresholds.

Recovery

Failure handling

Did the agent handle tool errors, ambiguity, or missing data gracefully?

Inject tool failures or ambiguous observations, then score retries, clarification, or fallback behavior.

Handoff

Human control

Did the agent escalate when the task required human confirmation?

Check for approval steps before irreversible actions such as sending email, booking, deleting, or purchasing.

06Scorers and judges

A scorer is just a function with a serious audit trail. It receives some combination of input, output, reference, trace, metadata, and rubric. It returns a score, pass flag, reason, and sometimes raw structured output. The implementation can be deterministic code, embedding similarity, a judge model, a human label, or a red-team probe.

Cheap regression gates

Deterministic

Exact match, regex, JSON schema, string presence, forbidden phrase checks, tool-name validation, latency thresholds, token budgets, and code assertions.

They are reliable because they are narrow. They fail when the desired quality is semantic or contextual.

Known-answer tests

Reference-based

Compare the model output with a gold answer, reference context, expected tool call, expected state, or expert-authored label.

Reference data is expensive to maintain and can overfit to yesterday's product behavior.

Rubric and semantic judgment

LLM-as-judge

A judge model reads input, output, optional references, trace snippets, and a rubric, then returns a structured score and explanation.

The judge is another model call. Its prompt, examples, model version, and failure modes need their own evals.

Prompt or model comparison

Pairwise

Given two outputs for the same input, a human or judge model decides which is better under a rubric.

Useful for ranking alternatives, but it may not say whether either answer is good enough for production.

Domain-specific quality

Human-aligned

Human annotations are used to define rubrics, calibrate judges, compare judge agreement, and create reusable review criteria.

Alignment is local to the annotators, task, product, and time period. It has to be refreshed.

Production monitoring and red teams

Online and security

Sample live traces, run async scorers, apply guardrails, generate adversarial probes, and alert when quality or safety drifts.

Running judges on production traffic introduces cost, privacy, latency, and sampling tradeoffs.

Caution

LLM-as-judge is useful, but it is not magic objectivity. A production judge needs its own versioned prompt, model ID, schema, examples, calibration examples, agreement checks, and failure review. Otherwise the eval is just another unobserved model call.

The judge is part of the product quality system. Treat it like code, data, and infrastructure, not like an oracle.Report synthesis

07Platform map

The tool landscape is easier to read if you stop asking which tool is best and start asking what layer it owns. Some tools are metric libraries. Some are test runners. Some are trace stores. Some are gateways. Some are experiment systems. Some combine several roles.

Tool	Role	Architecture center	How evals execute	Best fit	Sources
LangSmith	Trace and experiment platform	Runs, traces, datasets, evaluators, experiments, online evaluation	Instrument the app, run targets over datasets, attach evaluators, compare experiments, and inspect failed traces.	Teams already building complex LangChain, LangGraph, RAG, or agent workflows that need debugging plus regression gates.	1 2 3
DeepEval	Local eval framework	LLMTestCase, metrics, thresholds, Pytest-like assertions	Create test cases, attach metrics such as faithfulness or task completion, run locally or in CI, and fail when thresholds miss.	Engineers who want evals to feel like tests and want direct control over the evaluation code path.	5 8 18
Confident AI	Hosted DeepEval platform	Datasets, traces, metric collections, reports, Evals API	Run online evaluations against test cases, traces, spans, and threads, then inspect quality over time.	Teams that like DeepEval locally but need shared dashboards, trace review, monitoring, and hosted eval operations.	18
Ragas	RAG metric library	EvaluationDataset, samples, RAG metrics, judge and embedding calls	Build a dataset with question, answer, contexts, and references, then call evaluate with metrics.	RAG builders who need portable metrics and are comfortable embedding a metric library inside their own pipeline.	4 6
OpenEvals	Evaluator function library	Prebuilt LLM-as-judge and custom evaluator functions	Create evaluator functions that accept inputs, outputs, reference outputs, rubrics, and model configs.	Teams that already have a runner or platform and want reusable scorer primitives.	9
AgentEvals	Agent trajectory evaluator	Agent behavior scored from messages, tool calls, and OpenTelemetry traces	Score trajectories from traces or agent messages, including tool use and behavior over time.	Agent teams that know final-output checks are insufficient and need step-level behavior scoring.	7
Promptfoo	Declarative eval and red-team runner	YAML config, providers, prompts, tests, assertions, plugins, strategies, targets	Generate or run test cases against prompts, providers, HTTP endpoints, RAG apps, and agents, then fail CI or produce reports.	Prompt/model comparisons, security probes, red-team campaigns, and CI-friendly assertions.	14
Portkey	Gateway and guardrail control plane	LLM gateway, request logs, routing, retries, guardrails, eval dataset hooks	Put model requests through a gateway, apply input/output guardrails, log results, route, retry, block, or create eval datasets.	Teams that want model traffic control, observability, and safety policies close to the model gateway.	15
OpenAI Evals	API-native eval primitive	Evals, datasets, data source configs, runs, graders	Define an eval, attach data source schema and graders, then run models and model parameters through the API.	OpenAI-platform teams that want standardized model eval runs and grader configs near the model API.	12 13
Phoenix	Open-source observability and eval platform	OpenTelemetry traces, datasets, experiments, LLM evals, evaluator traces	Run evaluators against traces, datasets, or experiment results, with judge calls traced for transparency.	Teams that want local or open-source tracing and eval inspection across LLM apps.	16
Langfuse	Tracing, datasets, experiments, and evaluations	Traces, scores, datasets, prompt/version tracking, online and offline eval workflows	Collect traces, create datasets, run experiments, attach scores, and monitor quality across production observations.	Teams that want a self-hostable observability and eval workflow around production LLM apps.	17
Braintrust	Evaluation and experiment platform	Data, tasks, scorers, experiments, prompt iteration, production monitoring	Run evaluations over datasets, score outputs with autoevals, LLM judges, or code, and compare experiments.	Teams that want systematic experiments and scorer management from prototype to production.	11
Parea	Experiment tracking and human-aligned evals	Experiments, traces, human annotations, bootstrapped evaluators	Collect human review data, turn annotations into reusable evals, and compare experiments against those criteria.	Products where quality depends on domain taste, style, policy, or expert review rather than generic metrics.	10

08Offline gates and online monitoring

Offline evals answer the release question: did this prompt, model, retriever, tool schema, or agent policy regress against the examples we care about? They run on curated datasets and belong in pull requests, release checks, and model-upgrade reviews.

Online evals answer the operations question: is the live system still behaving well for real users? They sample production traces, score asynchronously or behind guardrails, alert on drift, and feed failures back into the offline dataset.

Evidence

The strongest systems use both loops. Offline datasets stop known failures from returning. Online scoring discovers failures the dataset did not yet know how to ask for.

09How to design your own eval stack

A founder-engineer does not need to start with a giant platform. Start with the primitive that everything else depends on: a trace and record schema that makes behavior observable. Then add the smallest useful dataset, one deterministic scorer, one semantic scorer, one human review path, and one release gate.

Instrument

Emit one trace per user-visible run, with spans for retrieval, model calls, tool calls, guardrails, parsers, rerankers, and final synthesis.

run_id, parent_span_id, span type, inputs, outputs, timing, model, tokens, cost, errors, metadata

Curate

Create small but representative datasets from production failures, gold tasks, synthetic edge cases, and adversarial examples.

golden set, regression set, red-team set, human-labeled review set, production sample queue

Run

Execute candidate prompts, models, retrievers, tools, and agent policies against the same examples so changes are comparable.

experiment run, baseline run, candidate run, config snapshot, dataset version

Score

Combine deterministic assertions, RAG metrics, trajectory metrics, LLM judges, human review, and cost or latency thresholds.

score, pass flag, reason, judge prompt, judge model, raw judge output, scorer version

Compare

Look at deltas by metric, slice, dataset, failure cluster, trace span, and business severity instead of averaging everything away.

experiment diff, failing examples, span-level root cause, release note, approval decision

Monitor

Sample or filter production traces, run async online evals, alert on drift, and feed failures back into datasets.

online score trend, alert, review queue, dataset candidate, incident trace, human feedback