Does a single benchmark score actually predict agent readiness?

Multiple 2026 findings converge on the same methodological claim, each from a different angle: agent capability cannot be evaluated as a scalar. It decomposes into separable axes, and benchmarks that score one axis produce rankings that fail to predict performance on the others.

Do phone agents succeed at all three critical tasks equally? is the cleanest demonstration. Across five frontier models on 300 phone-use tasks, the three properties are statistically distinct — no single model dominates all three. Models that win on task success may lose on privacy compliance because they complete tasks by overfilling personal data. Models with mediocre success may have better privacy compliance because they stop at minimal disclosure. The single-axis "task success" benchmark produces rankings that get reshuffled the moment a second axis is added.

Do short benchmarks predict how models perform over long workflows? adds a temporal axis. Models that perform comparably on single-turn benchmarks diverge dramatically by relay round 25 — different degradation curves, different decay characteristics, different failure modes. Length-of-interaction is its own axis; benchmarks that hold it short cannot characterize what happens long.

Are LLM and agent benchmarks really measuring different things? surfaces yet another axis: operating mode. A model's "LLM benchmark" performance (single-turn completion) and its "agent benchmark" performance (multi-step tool use) measure different operating regimes of the same artifact. The capability has at least two numbers, and which one matters depends on deployment context. Bifurcating the benchmark literature treats them as separate models; they are the same model in different modes.

Why do capable AI agents still fail in real deployments? adds an environmental axis. Even when an agent's intrinsic capability is high, deployment success requires ecosystem properties — value-generation alignment, personalization, trust, social acceptability, standardization — that capability benchmarks do not test. The benchmark is necessary but not sufficient.

Why do AI agents fail at workplace social interaction? grounds the abstract argument in deployment data. The benchmark-to-real-world gap is enormous — frontier models that score impressively on agent benchmarks fail 70% of real workplace tasks. The benchmarks and the deployment are not measuring the same capability.

Together these argue for a structural change in how agent capability is reported.

The current practice of ranking models by a single score is, at best, useful for narrow deployment matched to the benchmark's axis. For broader deployment, single-axis rankings are actively misleading. A capability vector with at minimum 4-5 axes — task success, privacy compliance, long-horizon retention, mode-shift behavior, ecosystem readiness — is the right unit of evaluation.

For benchmark designers, this argues that joint evaluation should be the default rather than a research add-on. A benchmark suite that reports five axes and refuses to collapse them to a single number forces the consumer to consider which axis matters for their deployment. Single-number rankings make the decision invisible.

For agent developers, this changes the optimization target. Optimizing for single-benchmark success produces models that win the benchmark and lose deployment. Optimizing for multiple axes simultaneously produces models that might score middling on any one but are robust across the deployment regime. The vector framing changes which model is "best" depending on the use case — which is what production deployment requires anyway.

The pattern extends beyond agents. Does preference tuning actually reduce the diversity of model outputs? shows the same dynamic for diversity evaluation: raw diversity and effective diversity diverge, and which metric matters depends on use case. Can post-training objectives preserve reasoning style alongside correctness? shows it for post-training objectives: answer correctness is one axis, reasoning style is another, and they can move opposite directions. The "vector not scalar" pattern recurs.

The deeper observation: capability has always been multi-dimensional. Single-number evaluation is a methodological convenience that compresses out structure the deployment context cares about. The compression is invisible while the benchmark axis aligns with deployment needs and visible (as deployment surprise) when it does not.

Source: synthesis across Flaws, Assistants Personalization, Agents, Autonomous Agents