AI agent benchmarks are becoming systems tests as harness design moves the score

Single-turn language benchmarks could often approximate a direct mapping from prompt to model output. Agent evaluations add a software layer between the benchmark and the model. That layer decides what the model sees, which tools are exposed, how tool results are represented, when history is truncated or compacted, whether errors are retried, how plans are persisted and when the run ends.

That distinction stopped being academic when OpenAI reran ARC-AGI-3 with a different harness. The published comparison used the same GPT-5.6 Sol model but changed how reasoning state and long context were handled. The reported score rose from 13.3% to 38.3%, while output-token use fell by roughly sixfold. A result that large cannot be described responsibly as a property of the underlying model alone.

The useful unit of comparison is therefore a versioned agent system. A model-only leaderboard can still answer a narrow research question if every other component is genuinely controlled, but product capability requires reporting the harness that makes the model act.

Long-running agents repeatedly encounter the same state-management problem: the trajectory grows faster than the useful working set. A naive harness eventually drops early observations or lets irrelevant history consume the context window. A better harness preserves durable facts, compresses older work and keeps recent high-value state available to the model.

ARC-AGI-3 exposes this sharply because the agent must explore an unfamiliar environment and infer its rules over time. If the harness discards private reasoning after each action or truncates early history, the model may repeatedly relearn the same structure. Retaining reasoning and compacting context changes the effective memory available to the policy.

This means context engineering should be disclosed like any other evaluation setting. Two systems that use the same model but different memory policies are not equivalent experimental conditions.

Agents execute code, install packages, build repositories and manipulate files. Those actions make infrastructure part of the task. Anthropic measured this directly on Terminal-Bench 2.0: with the same model, harness and tasks, the least- and most-resourced configurations differed by six percentage points. Infrastructure errors fell from 5.8% in the strictest setting to 0.5% without the same cap.

There is a point where extra resources stop merely preventing crashes and begin changing what strategies are feasible. More memory may permit a full build instead of a targeted workaround. Longer timeouts may permit a slower search strategy. A benchmark that hides these limits can unintentionally reward infrastructure generosity.

For fair model research, use a common envelope. For product evaluation, a cost-performance frontier is often more informative: report the score obtained at a stated compute, memory, time and monetary budget.

Coding benchmarks depend on tests as executable ground truth, but tests are not automatically correct. OpenAI audited SWE-Bench Pro after frontier coding performance climbed rapidly and found a large fraction of tasks with problems such as overly strict tests, underspecified prompts, weak coverage or misleading issue descriptions. Its combined auditing work led to an estimate that roughly 30% of the benchmark was broken.

The earlier SWE-bench Verified audit revealed a similar problem from another direction. OpenAI stopped reporting the benchmark after finding substantial task flaws in the audited subset and evidence that frontier models had seen some benchmark material during training. Once models approach the ceiling, a rising share of apparent failures can belong to the benchmark rather than to the model.

Evaluation pipelines therefore need benchmark QA as an ongoing operation. Failure analysis is not optional cleanup after scoring; it is part of maintaining the measurement instrument.

An agent can produce a persuasive explanation while leaving the repository broken. It can also complete the task correctly while describing its work poorly. Anthropic’s evaluation guidance makes the distinction explicit: outcome grading should inspect the external state created by the agent rather than trusting its final text.

For coding, that usually means tests, repository diffs, build state and task-specific invariants. For browser or enterprise agents it may mean records created, forms submitted, permissions respected or database state changed. The grader should be as independent as practical from the same model that performed the work.

This is also why trajectories should be retained. A pass/fail bit cannot tell whether the system succeeded through a robust plan, a lucky guess, an unsafe shortcut or a grader loophole.

HarnessOpt-Bench, posted in August 2026, treats the surrounding agent software as an explicit optimization target. Its experiments ask frontier models to improve prompts, tools, memory and control flow under held-out evaluation, with resource use metered in a trusted execution environment. The work is early, but it captures a direction already visible in production: teams increasingly optimize the system around the model rather than waiting for a larger model release.

This creates a new source of benchmark overfitting. Repeatedly tuning a harness against the public test set can produce a system that looks excellent on one benchmark and transfers poorly. Held-out tasks, versioned optimization budgets and cross-benchmark testing become necessary once harness engineering is itself part of the search loop.

A useful metric is portability: how much of a harness uplift survives when the model, repository family or task distribution changes.

Software agents increasingly read issue text, repository files and web content that may be untrusted. A benchmark focused only on whether the final tests pass can reward a system that follows malicious instructions along the way, leaks secrets or exceeds its intended permissions.

IssueTrojanBench illustrates the problem with adversarial issue content across coding agents and multiple model families. The authors report that 66.5% of malicious issues in their evaluation penetrated all tested guardrails. That number should be treated as a research result tied to the benchmark setup, not a universal failure rate, but it makes the measurement gap concrete.

Agent evaluation should therefore pair task success with policy and security outcomes: forbidden tool use, secret access, network egress, destructive actions and instruction-source violations. An unsafe success is not equivalent to a safe success.

Leaderboards remain useful for screening, but procurement and release decisions should run the candidate model inside the real production harness. That means the actual tools, permissions, memory policy, context compaction, retry limits, resource envelope and acceptance tests.

OpenAI’s recent scientific-computing field report points to the operational boundary: implementation can accelerate dramatically, but the bottleneck moves toward validation. Across its reported projects, agents were most reliable when a measurable external target or reference implementation existed. Confident but wrong outputs still appeared, especially around last-mile edge cases.

The practical lesson is not to discard benchmarks. It is to use them as instruments with known calibration. Keep a stable reference harness for longitudinal model comparisons, a production harness for deployment decisions, and explicit ablations when a score changes enough to affect a release.

The most useful near-term change would be richer result cards: model version, harness commit, tool set, memory and compaction policy, resource envelope, retry budget, cost, grader revision, task exclusions and confidence interval. Without that record, a point score is hard to reproduce and easy to overinterpret.

Component-level benchmarks are also likely to grow. AgentCompass separates benchmark, harness and environment; HarnessOpt-Bench focuses on optimizing the harness itself; new safety evaluations probe whether apparently capable agents can resist malicious task content. These are different questions, and combining them into one universal score would lose the information teams actually need.

Watch for whether major benchmark maintainers begin publishing harness sensitivity and resource sensitivity as standard fields. If they do, “which model won?” will become a less useful question than “which complete system performed best under the constraints we care about?”