When AI helps scientists earn far more citations while science's topics shrink, can citation counts still measure impact?

This explores whether citation counts are a trustworthy main measure of research impact. The collection doesn't hold a classic bibliometrics debate. What it shows instead is citations being used more and more as the *ground truth* for other things, while evidence builds up that the count can rise even as the science underneath gets narrower. The real question, then, is less 'are citations good?' and more 'what happens when everything, including AI, is tuned to them?'

The sharpest warning comes from studies of AI-augmented researchers. Scientists who use AI publish about three times as many papers and receive 4.8 times as many citations. Across science as a whole, though, the range of topics shrinks by 4.63% and collaboration between researchers drops by 22%, because work drifts toward data-rich problems people are already crowded around Does AI help individual scientists while narrowing scientific focus?. By the citation metric this looks like a boom. By the measure of whether science is exploring new questions, it looks like a contraction. A metric that can't tell these two apart is a risky one to put first.

The opposite move is also happening: citations are becoming the target that AI systems learn from. One project trained a model on 700,000 citation-matched pairs of papers to learn 'scientific taste', meaning it predicts which ideas will have high impact Can models learn what makes research worth doing?. Another study found that authors' private rankings of their own papers predicted later citations better than ICML peer-review scores did Can authors rank their own papers better than peer reviewers?. Both results are interesting, but both quietly treat 'gets cited' as the definition of 'good'. If citations are biased toward fashionable or crowded topics, a model trained on them learns that bias and calls it taste. It may then help produce the same narrowing the first study measured.

There is also evidence on why citation counts are noisy in the first place. In AI search tools, users preferred answers with more citations almost as strongly when the citations were irrelevant (β=0.273) as when they were relevant (β=0.285). The sheer number of citations acts as a trust signal, separate from what the citations contain Do users trust citations more when there are simply more of them?. Attention can also spread before any quality check. MIT's case shows an unreviewed arXiv preprint shaping the debate long before the institution disowned it Can unreviewed preprints shape scientific debate before peer review?. Meanwhile, a review of 230 publications describes a coupled arms race in which AI scales up paper production, automates evaluation, and invites manipulation and evasion Does AI create a coupled arms race in research production and review?. Any count that can be pumped up will get pumped up.

The same proxy problem appears in AI research benchmarks. Gains measured under a fixed evaluation budget don't show whether actual research costs fall Do fixed-budget efficiency gains translate to real research progress?. The pattern holds in both places: a convenient number stands in for 'progress' until optimizing the number pulls away from the goal. The corpus suggests citations are useful as one signal among several, for example alongside measures of topic diversity or models of how discoveries actually happen Can predicting scientists improve discovery forecasts?. As the main measure, they risk rewarding the very concentration that AI is already speeding up. The gap: the collection has little on alternative impact measures themselves, so it can say clearly what goes wrong but not what should replace citations.