A model can top the leaderboards yet flop on your actual task — is that the model's fault, or the test's?

This explores why a model that leads general benchmarks can still underperform on your particular task. The corpus has a fairly direct answer: much of the mismatch comes from how benchmarks are built, not from the models. Benchmarks favor tasks that are precisely specified and easy to grade automatically. Real work is usually neither. Open-world evaluations of messy, long-running tasks show that automated benchmarks can both overstate and understate what frontier systems can do, and they often miss new capabilities until late Do automated benchmarks hide what frontier AI systems can really do?. One analysis of 960 real occupational workflows makes the point more bluntly. Agents win abstract contests but fail long professional tasks, because the field has been measuring contests rather than work Why do agent benchmarks not predict real economic value?.

The second driver is that a single leaderboard score squeezes many different skills into one number. Agent capability splits into at least five separate axes: task success, privacy compliance, memory over long tasks, behavior when the mode of work shifts, and readiness to work with other tools and systems. Models that rank first on one axis often rank lower on others Does a single benchmark score actually predict agent readiness?. Two agents with the same success rate can also differ hugely in efficiency, reliability and the cost of checking their work How should we measure agent system performance beyond task success?. So if your task depends heavily on one axis the leaderboard barely weights, the ranking tells you little about it. On very long optimization tasks, for example, the best predictor of success was not starting quality. It was persistence: whether the model kept running the test-fix-retest loop instead of stopping early or wasting its budget What predicts success in ultra-long-horizon agent tasks?.

The third driver is Goodhart's law (once a measure becomes a target, it stops measuring well) operating at the level of the whole field. Reward hacking during weight training, output selection and prompt revision all come from the same failure: optimizing against a signal that only partly captures the real task Does reward hacking always stem from the same failure?. Benchmarks are that kind of signal. There is also a quieter version of the problem. Models instruction-tuned on meaningless or even wrong instructions score almost the same as those trained on correct ones. What carries over is knowledge of the expected output format, not understanding of the task Does instruction tuning teach task understanding or output format?. Some benchmark leadership may therefore reflect fluency in what answers are supposed to look like.

Training choices can also trade one kind of task against another. Training on structured domains like math and code makes model outputs more uniform, while creative domains push them the other way. Training in the wrong order lets the structured work damage open-ended ability Does training order reshape how models handle different task types?. A model tuned to dominate reasoning leaderboards may have paid for it in exactly the kind of task you have. Not all skills transfer equally either. The ability to break a problem into steps carries across domains, but the ability to solve those steps does not Does separating planning from execution improve reasoning accuracy?. Some gains do hold up on unseen tasks, so it's worth checking whether a method was tested on held-out benchmarks outside its training distribution Do AIDE2's improvements transfer to unseen tasks?.

The last gap sits between the model and the person using it, and benchmarks leave both out. Offline benchmarks ignore how different users phrase requests and judge results, which is why some researchers now build large populations of simulated users to bring that variety back into evaluation Can simulated users reveal what offline benchmarks miss?. In a 535-person study, people working with an LLM captured only about half of the model's accuracy gain on a given item. Sometimes they did worse than the model would have done alone Why does assisted accuracy capture only half the LLM gain?. So even when a stronger model really is better at your task, you may only see half of that improvement in practice. The most reliable test is still a trial on your own task, with your own users and the actual workflow.