26 Feb 2026 AI Engineering5 min read

Agent Evals in CI: Regression Gates for Trajectories and Tool Calls

Agents need different regression gates from deterministic software. We show how tool calls, trajectories, offline datasets, production traces, and cost budgets form a tiered CI strategy that is fast enough for pull requests and robust enough for production.

Why agents break classical CI

An agent works as a loop: the model receives a task, calls tools, observes results, and chooses the next step. Even with identical input, it rarely follows the same path twice. Classical tests assume a fixed contract — one expected output per input. That contract is too narrow for agents. A prompt change can repair one scenario and silently degrade three others without any unit test failing.

τ-bench, published in June 2024, shows the size of that gap. The benchmark paired function-calling agents with simulated users and real APIs. GPT-4o solved about 61 percent of retail tasks in a single trial. Require eight runs of the same task to succeed — pass^8 — and the rate falls below 25 percent. The bottleneck is therefore not peak capability but repeatability. That is what CI must expose.

A change lands — prompt, model or retrieval. 1/4

The trajectory is the unit under test

A trajectory is the sequence of tool calls an agent makes, including the arguments it generates, plus the state it leaves behind — files, database rows, messages. Evals assert against this sequence, not only the final answer: was the right tool called, with the right arguments, before the side effect happened?

Two complementary granularities work in practice. Single-step evals stop the agent after one decision and inspect that step — cheap, fast, and precise. LangChain reported in December 2025 that about half of its agent test cases followed this pattern. Full-turn evals run the entire loop and score the final response, trajectory, and state changes as one outcome.

One caution: strict sequence matching over-constrains. Agents reach correct results through valid paths the test author did not anticipate. Reserve exact-order assertions for cases where order is a correctness or safety requirement; elsewhere assert that a tool was called, not when. Open-source evaluators such as agentevals (February 2025) support both modes: strict trajectory matching and judge-based trajectory scoring.

Regression gates on every pull request

A gate works only while the team does not route around it, and slow gates are eventually bypassed. An LLM-judge suite that takes ten minutes per push will lose required status within weeks. We therefore tier checks by latency and cost: deterministic checks on every commit, judge-based scoring at merge, and the complete dataset overnight.

The delta matters more than the absolute score. A suite that falls from 96 to 91 percent has regressed even if it still clears a 90 percent threshold. Store the main-branch baseline, compare every pull request against it, and block once the drop exceeds a defined margin. Run non-deterministic cases repeatedly — for example, three trials with majority pass — so sampling noise is not mistaken for regression.

Tier	Trigger	Checks	Merge gate
Deterministic	Every commit	Schema validity, expected tool calls, argument checks, cost ceilings	Yes — 100 percent required
LLM-as-judge	Merge to main	Rubric scores, refusal behavior, trajectory quality	Yes — threshold with repeat runs
Full regression	Nightly	Complete dataset, cross-model comparison, cost audit	No — alert only

Offline datasets and online judges

Offline evals run against a curated dataset with reference outputs. Correctness is defined per case; deterministic graders check it. This is where regression gating lives. The dataset does not need to be large: Anthropic's guidance from January 2026 is that 20 to 50 tasks drawn from real failures are a sufficient start, because early changes have large effect sizes.

An offline dataset can never fully anticipate real users. Online judges add a second perspective: an LLM continuously scores sampled production traces against a rubric without requiring reference answers. LangSmith shipped multi-turn evals for whole conversations in October 2025; Langfuse runs judge pipelines over stored traces. The value lies in the feedback loop: a trace flagged online becomes a new offline case, preventing the same defect from returning after it is fixed.

Judges are measurement instruments and need calibration. Score at temperature zero, write rubrics that name explicit conditions, and check judge output against human labels before trusting it. An uncalibrated judge produces a number, not a measurement — and judges drift just like the agents they score.

Cost per run as a first-class metric

Agent evals consume real tokens. The τ-bench authors reported roughly $0.61 per task and trial — $0.38 for the agent, $0.23 for the simulated user — with 96 percent of the agent's cost in input tokens. A suite of 100 cases with three repeats is a measurable line item on every pull request. Track it per run, or the suite gets budget-cut before the quarter closes.

Token accounting is standardizing; cost accounting is not. The OpenTelemetry GenAI semantic conventions define token-usage attributes and metrics (gen_ai.usage.input_tokens, gen_ai.client.token.usage), still in Development status as of February 2026. A standardized cost metric does not exist — you compute cost from token counts and a maintained price table. Then use it: a change that doubles tokens per task is a regression even when quality scores hold. Cost ceilings belong in the deterministic tier.

Eval-driven development

Eval-driven development reverses the usual order: write the evaluation before the capability. Its initially low pass rate does not describe project failure; it defines the target to reach. When a new model ships, a single suite run shows which cases improved and which broke. Anthropic describes features that work only "well enough" today as bets on future models; capability evals make those bets testable.

Two disciplines keep the loop honest. First, environment hygiene: every trial starts from a clean state, and external APIs are recorded and replayed, because shared state produces flaky evals that measure nothing. Second, read the transcripts. A failing grade does not distinguish an agent mistake from a grader rejecting a valid solution. Only the transcript does.

Outlook from February 2026

Three developments look likely from here. First, trajectory data becomes portable: OpenTelemetry's GenAI conventions already model tool-call spans, while MCP standardizes the tool interface. Eval harnesses will therefore be able to consume traces from different runtimes. Second, cost enters the conventions as a first-class signal. Third, judge calibration moves from ad-hoc scripts into the platforms.

The durable asset is therefore not the public benchmark; it saturates. The lasting value sits in the private, product-specific suite that grows with every production failure. We treat it as a deliverable with the same status as the agent itself: versioned beside the code, gating every merge, and owned by the same role that owns the specification. Models will change; the evals preserve the knowledge of how the system is expected to behave.