Do interactive evaluations actually solve the benchmark comparison problem?

The seductive promise of interactive evaluation is that richer evidence solves the problems of response-centered benchmarks. The position paper resists this. Its analysis shows that longstanding evaluation challenges — comparability, reproducibility, the validity of the evidence-to-judgment mapping, what claims a score actually supports — reappear at the trajectory level rather than disappearing. Scoring a path instead of an endpoint does not escape the core difficulty of evaluation; it relocates it into a higher-dimensional space where it is, if anything, harder to pin down.

This is why the paper frames the situation as a question demanding design, not a solution already in hand. A trajectory admits many scoring choices, and different interactive benchmarks make incompatible ones, so their results are not interchangeable — the same fragmentation that response benchmarks eventually had to standardize away, now recurring with more degrees of freedom. Process quality, recoverability, and coordination are genuinely informative, but each introduces its own version of the old questions: what counts as evidence, how is it aggregated into a judgment, and what does the resulting number license you to claim?

Why it stays open: the honest reading is that interactive evaluation buys richer evidence at the cost of reintroducing every hard problem at a new scale. The field's task is therefore not to adopt the format but to build the protocols, robustness tests, shared infrastructure, and reporting standards that make trajectory scores interpretable — work that is unfinished. Treating the new paradigm as a fix would repeat the mistake; treating it as a design problem is the corrective the paper argues for.