Do AI-detection methods agree with each other, or does a detector that wins on one model fail on the next?

This explores how the different instruments used to detect AI compare with each other, and whether they give consistent answers from one model or setting to the next. The short version is that the instruments disagree a lot. A difference that a machine can measure easily can be completely invisible to a person. A detector that beats an expensive monitor on one model can lose to it on the next. One caveat first: the collection has no head-to-head comparison of commercial AI-text checkers across websites or apps. If that's what you meant, this is a gap in the corpus.

The sharpest split is between measurement and perception. AI-written text differs from human text in ways you can measure: across six dimensions of vocabulary variety, the gap shows up consistently over multiple models Can humans detect AI text if machines can measure it?. Yet human judges, trained linguists included, can't reliably tell the difference. Newer models drift *further* from human statistics while getting *harder* for people to spot. A 30-study review finds the same pattern for text, images, and voice: people detecting AI content score around coin-flip accuracy, and they haven't improved as the generators have Can people reliably spot content made by AI?. So 'detector-based' carries real weight in the question. Statistical detectors and human eyes aren't weaker and stronger versions of the same tool. They pick up different things.

A second split appears when the target is AI *behavior* rather than AI text, for example catching an agent that games its reward instead of solving the task. One option is a difference-of-means vector. This is a cheap probe that reads signals already present in the model's internal activations, so it costs almost nothing extra to run. The other option is a separate LLM monitor that reads the agent's output. At matched false-alarm rates, the probe caught 3.1% more hacks than the monitor on one model (Kimi K3) and 7.9% fewer on another (GLM 5.2) difference-of-means-vectors-are-similarly-effective-to-llm-monitors-but-virtually. That is the clearest cross-model lesson in the corpus: neither approach wins everywhere. Which one does better depends on the model being watched, so a detector validated on one model shouldn't be assumed to carry over to the next.

When detection becomes judgment, the design of the judge matters enormously. An agent that actively gathers evidence before ruling cut 'judge shift' (how far its verdicts drift from the correct ones) from about 31% for a plain LLM judge to 0.27%. But its memory component let errors cascade, a reminder that more elaborate detectors bring new ways to fail Can agents evaluate AI outputs more reliably than language models?. A related idea is to stop trusting a single final score and instead certify *how* a result was produced, using recorded infrastructure evidence Can infrastructure evidence replace terminal scores in benchmark validation?.

The thing you may not have expected to want to know: the instruments are fragmented, and nobody has yet built one that captures the whole picture. Work on whether AI errors stay visible and fixable finds only partial measures. There are model-side signals like reasoning disclosure, incident counts for containment, and rollback timing for recovery, and nothing that bridges them or accounts for the humans and institutions in the loop How can we measure whether AI errors stay visible and recoverable?. The differences between detection methods aren't just noise to average out. Each method sees a different slice of the problem, so the honest approach is to combine several detectors and report where they disagree.