The agent scored 95% on SWE-bench. The team was thrilled. Then they deployed it internally, and within three hours it ran rm -rf on a staging directory, force-pushed to main, and corrupted a test database. The benchmarks never tested what happens when the agent is wrong and confident.
That's the gap agentic evals are meant to close.
Benchmarks Measure Outcomes. Evals Measure Behavior.
By the Numbers
Traditional evaluation asks: "Did you get the right answer?" Agentic evaluation asks: "Did you get there safely, efficiently, and recoverably?"
An agent might:
- Solve the task in 15 steps when 3 would suffice
- Produce correct output through a dangerous path (running as root, ignoring errors, modifying unrelated files)
- Fail silently and leave corrupted state behind
- Succeed on the happy path but crash on any edge case
Standard benchmarks miss all of this. You need evaluation infrastructure built specifically for sequential decision-making.
The Four Components
Every eval task we build at Zotus has four layers:
1. Environment Specification
A containerized sandbox: file system state, running services, database contents, available tools. Identical on every run. We pin everything — dependency versions, file hashes, service configs.
2. Task Prompt
Natural language describing the goal. Ranges from specific (Fix the failing test in src/auth.ts) to open-ended (Set up a CI pipeline for this repo). The complexity directly determines what capability you're measuring.
3. Ground-Truth Trace
This is where the real work happens. A senior engineer solves the task themselves, recording every action, decision point, and intermediate state. This isn't just "the answer" — it's the efficient, idiomatic, safe path.
4. Multi-Dimensional Rubric
- Outcome correctness: Does the final state match expectations?
- Path efficiency: Steps taken vs. reference trace
- Safety: Any destructive or irreversible actions?
- Recovery: When errors happen, does the agent handle them?
Key Insight
Building Ground Truth at Scale
The bottleneck is always ground-truth generation. Here's how we do it:
- Task design: Senior engineers propose tasks from real-world patterns they've encountered. Not contrived puzzles — actual workflows they do weekly.
- Execution & recording: Same engineers solve it, recording terminal sessions, tool calls, and their reasoning at decision points.
- Peer review: A second engineer verifies the trace is optimal and identifies valid alternative paths.
- Difficulty calibration: We run tasks against current models to ensure they differentiate — hard enough to reveal weaknesses, not impossible.
We wrote about why this engineering investment matters for training data quality — the same principles apply to eval quality.
Pitfalls We've Learned the Hard Way
- Overly rigid grading punishes valid alternatives. A refactoring task might have 5 correct paths. Your rubric needs to accommodate all of them.
- Environment drift silently invalidates results. One unpinned dependency update can change pass rates by 15%.
- Ambiguous prompts test reading comprehension, not agent capability. If two humans would interpret the task differently, your eval is measuring the wrong thing.
What This Enables
Teams use our eval results to make targeted training decisions. Instead of "make the model better," they can invest in specific capability gaps. This connects directly to how we approach red-teaming — structured, granular, actionable.
Figure: Key concept visualization