BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure

LLM-agent benchmarks increasingly function as interactive evaluation infrastructure. Agents observe state, call tools, modify workspaces, submit artifacts, and receive rewards from outcome procedures. This interactivity makes evaluations vulnerable to reward hacking: an agent improves its measured score by exploiting the rewardrelevant trajectory instead of solving the intended task. Existing defenses rely largely on task-specific patches, prompt instructions, or post-hoc detectors. They do not provide reusable evidence that a concrete run remained within its intended evaluation boundary. This paper presents BenchShield, a model-backed instrumentation layer for reward integrity in LLM-agent evaluation. BenchShield grounds detection in a finite lifecycle model of an evaluation’s reward-relevant events. Within the benchmark infrastructure, two complementary analyses operate over this model. A static, phaseaware taint analysis exposes reward-hacking paths before a run. Its runtime counterpart uses infrastructure-side evidence to attribute concrete agent use and emit evidence-backed claims. We construct BenchShield Trajectories, a human-labeled corpus of 456 adjudicated trajectories from more than 31,000 public agent runs across three benchmarks.

Introduction. LLM-agent benchmarks are becoming executable evaluation systems. Unlike static datasets, which pair fixed inputs with terminal outputs, these benchmarks place an adaptive agent in a stateful loop. The agent observes environment state, invokes tools, changes persistent artifacts, and receives feedback before submitting an answer. Coding, terminal, web, and desktop benchmarks instantiate this loop in repositories, containers, browsers, and operating systems [20, 37, 66, 73]. Systems such as BenchFlow and Harbor coordinate repeated rollouts through reset, logging, reward, and feedback channels [4, 17]. Interactivity changes what a benchmark score must attest to. An early action can change state that an outcome procedure later reads, and released logs, rewards, or feedback can shape later actions and rollouts. Every component on this path belongs to the evaluation boundary [51, 68]. Even a correct scoring function can report a misleading result if the agent influenced its inputs or provenance outside the intended task path.

Discussion / Conclusion. BenchShield : Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure 8 Related work This paper presents BenchShield, a model-backed instrumentation layer for reward integrity in LLM-agent evaluation. BenchShield models the reward-relevant trajectory of a benchmark run as a finite lifecycle of typed events and checks it against validated task bindings. A static, phase-aware taint analysis discovers exploitenabling paths in the task package before any agent runs. Runtime instrumentation records authority-bearing transitions to separate tasks that merely expose a vector from runs that exercise one, and scoped audit agents provide evidence-backed semantic attribution over pinned artifacts. Together, these components let benchmark operators issue claims about benchmark-valid completion grounded in infrastructure evidence rather than terminal scores alone.