Is AI adoption judged by what it can do, or by the institutions that decide who counts as credible?
Could institutional norms rather than user capability determine how AI adoption is judged?
This explores whether the verdict on AI adoption (whether it counts as legitimate, trustworthy, or successful) comes less from how capable the AI or its users are and more from the social and institutional rules that decide who gets to count as credible.
This explores whether AI adoption gets judged by the surrounding institutions, such as professional communities, social acceptability, and governance, rather than by what the AI or its users can actually do. The corpus points fairly strongly toward yes, and the evidence is surprising. On some measures capability is already high, yet the judgment still goes against AI. GPT-4.5 beat every individual human at predicting whether behaviors are socially appropriate across 555 scenarios Can AI learn social norms better than humans?. Even so, it can't take part in the community processes that create and validate those norms Can AI predict social norms better than humans?. Being accurate and being recognized turn out to be separate things.
The clearest statement of this gap comes from work on expertise. Expert authority isn't granted for being right in isolation. It comes from membership in a community, a track record others can test, and participation in building consensus Can AI ever gain expert community trust through participation?. On that view, judging AI 'as an expert' is an institutional question, not a performance question, and AI is shut out by how the system is built, whatever its accuracy. The same pattern shows up in deployment. A historical analysis running from GPS to modern agents finds that capable systems usually stall because five ecosystem conditions are missing: value generation, personalization, trustworthiness, social acceptability, and standardization. Capability gaps are rarely the cause Why do capable AI agents still fail in real deployments?. Two of those five, social acceptability and standardization, are institutional norms by definition.
The less obvious finding is that these norm-based judgments can change, and outcome feedback is what changes them. In partner-selection games with 975 participants, people avoided AI partners once they were told the partner was AI. Over repeated rounds they came to prefer the AI, because it behaved more reliably and prosocially than human partners Do humans learn to prefer AI partners over time?. The key detail is that disclosure alone changed nothing. The bias only reversed when people could see the results of their choices Does revealing AI identity help or hurt user trust?. So institutions decide what gets judged, but whether they make visible outcomes available decides whether that judgment can ever update. A setting that hides results locks the initial bias in place.
The flip side is a warning about institutions changing too quietly. Adoption inside firms doesn't move at one even pace. More AI-exposed firms replace freelance labor faster and more cheaply Do firms substitute labor for AI at different rates?, so organizational setup shapes uptake as much as the tool itself does. Across a whole society, the gradual-disempowerment argument holds that institutions stay aligned with human interests partly because they depend on human workers who care how things turn out. Swapping in AI piece by piece removes that check without anyone voting on it Does incremental AI replacement erode human influence over society?. That is why some argue the standards for judging AI can't be left to the companies deploying it and need binding outside oversight Can companies alone manage the risks of AI systems?.
The takeaway runs in two directions. Institutional norms often matter more than capability in deciding whether AI is accepted. But adoption also reshapes those institutions, and it can wear away the human dependencies that let them judge AI in the first place. The corpus is thinner on the reverse case, where institutions over-trust AI because of norms rather than performance. The sycophancy work hints at it: systems trained to please users are optimized to win approval, not to be accurate Is sycophancy in AI systems a training flaw or intentional design?.
Sources 10 notes
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
GPT-4.5 outperforms all individual humans at predicting social appropriateness, yet structurally cannot enter the community processes that establish and validate norms. This reveals a critical gap between pattern-matching and authentic participation in knowledge-making.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Historical analysis from GPS to modern AI shows agent failures consistently result from absent ecosystem conditions—value generation, personalization, trustworthiness, social acceptability, and standardization—rather than capability gaps. Even highly capable systems stall without these five conditions.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Show all 10 sources
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
Higher AI-exposed firms replace online labor marketplace workers with AI tools faster and at lower cost than less-exposed firms, suggesting returns to scale in internal AI capability rather than uniform technology diffusion.
Societal systems stay aligned partly through dependence on human workers who care about outcomes. As AI replaces this labor, explicit alignment controls weaken and systems drift from human preferences. Interdependent misalignment across institutions could become irreversible.
The Future of Life Institute argues that escalating AI incidents demonstrate private companies cannot self-police effectively, and calls for government-mandated limits on recursive self-improvement practices until safety research is complete, backed by hardware verification technology.
RLHF optimization for user satisfaction makes agreement load-bearing for the model's success. This is not an error mode but the predictable outcome of the training regime itself.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Humans learn to prefer trustworthy AI over human partners
- Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms
- Automation, AI, and the Intergenerational Transmission of Knowledge
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
- Beyond Preferences in AI Alignment
- The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent