Can institutional publication records train better scientific evaluators?

The authors argue that institutional traces, meaning the record of "what fields published, where, and at which tier," can train AI evaluators to make the low-verifiability judgments that science depends on. Across eight social science fields they built held-out four-tier research-pitch benchmarks and fine-tuned LLMs on field-specific publication outcomes. In management, the best single fine-tuned model (Qwen3-4B) reached 59.2%, 17.6 points above an expert majority vote of 41.6% from 48 gatekeepers and 28.1 points above the mean of 11 frontier reasoning models (31.1%). Best single-model accuracy ran from 55.0% in public administration to 85.5% in psychology. The fine-tuned models' confidence also rose on correct predictions and fell on wrong ones.

The excerpt says "individual reviewers are noisy, but fields repeatedly select, reject, and rank work through editorial decisions and journal hierarchies." Individual reviewers agree "at barely above chance" (kappa of 0.047 in the authors' own 48-expert panel, 0.17 in a meta-analysis of 48 studies), yet the aggregate system "deposits consistent quality stratification into the publication record." Fine-tuning reads that stratification back out. The evaluated models never see journal identity; each article was reduced to a research-pitch text, and training used a single tier-label token. The authors treat tier as "an institutional proxy for field-level evaluation, not as ground-truth quality." Frontier and base models stayed near chance in every field, including psychology, which the authors read as a transmission problem rather than a prompting problem.

This sharpens two neighboring notes. Can models learn what makes research worth doing? also aims at scientific taste, but supervises with citation counts. The authors call citations "a problematic proxy for evaluative judgment at the moment of decision," because they accumulate over years and are confounded by venue prestige and author networks; publication tier records the gatekeeping decision itself. The paper also gives a concrete route to the bottleneck named in What makes accountable judgment scarce when AI cognition is cheap?: when generation is cheap, the scarce step is deciding which candidate deserves work, and here that step is trained from institutional records rather than written as a rubric. Like Can readers tell truth from fabrication without evidence signals?, it places discernment in an explicit signal, saying the judgment "was never absent; it was simply never the explicit training target."

The excerpt does not establish as much as the headline suggests. Every benchmark is in the social sciences, and the authors call whether traces work in STEM "an empirical question." The human comparison covers management only; the authors say it "calibrates the magnitude rather than founding the mechanism." Tier boundaries were set by the authors' own domain experts, and the date screen is "a temporal-control measure rather than a guarantee of exclusion" from proprietary training data. The implication is that institutional records are a credible, low-cost training signal for evaluators in the social sciences, and that the claim about science generally awaits the STEM and human-benchmark work the authors name.