As AI gets better at research, what's left for human experts to actually do?
What role should human experts play as AI capability grows?
This explores where human experts fit as AI gets more capable: what they still do that AI can't, what their jobs turn into, and whether keeping them involved is a safety measure, a speed-up, or both.
This explores where human experts fit as AI gets more capable: what they still do that AI can't, and what their jobs turn into. The corpus doesn't land on "humans as a fallback for what AI can't do yet." It points somewhere more interesting. Some of what experts provide isn't a capability AI is missing. It's a social role that AI may never be able to occupy.
Start with the head-to-head data. On METR's research-engineering benchmark, AI agents score about four times higher than human experts when both get two hours. Give everyone eight hours and humans pull narrowly ahead. At 32 hours, humans lead by roughly 2× When do AI agents outperform human research experts?. Agents are sprinters that plateau, and people keep improving with sustained effort. That's one reason claims that automated AI research could squeeze years of progress into months deserve skepticism. Those forecasts assume that skill on small, checkable tasks carries over to long, consequential research, and that assumption hasn't been shown Could automated AI research compress years of progress into months?. A related argument holds that every major AI breakthrough so far needed humans to come up with new kinds of data and methods. On that view, human-AI "co-improvement" is both faster and safer than letting AI improve itself alone Can human-AI research teams improve faster than autonomous AI systems?.
The less obvious point is that expertise was never only about being right. One strand of the corpus argues that experts earn authority by taking part in a community: building a track record others can check, being challenged, and helping form consensus. AI can't enter that circle no matter how accurate it gets Can AI ever gain expert community trust through participation?. Expert judgment is also communicative. A good expert is always anticipating what a specific audience will find acceptable and credible, which makes AI's fluent, confident prose misleading in a particular way Can AI replicate the communicative work experts do?. So a practical rule follows: treat AI output as one piece of evidence to weigh, not a verdict that replaces your own reasoning. Stop deferring to it when the domain shifts, when bias shows up, or when new evidence arrives Should AI outputs replace or supplement human judgment?.
The obvious answer, "experts become reviewers of AI output," has a trap built in. Experts are already being pushed from producing knowledge to looking after AI-generated knowledge. That custodial role removes the arguing and testing that kept experts' knowledge honest in the first place Does AI reshape expert work into knowledge management?. At scale it becomes "epistemic hyperinflation": AI produces claims faster than people can check them, and the checking tools are increasingly AI-made too, so confidence in what we know erodes Can AI generate knowledge faster than humans can evaluate it?. Mollick offers the hopeful version. Deep knowledge, broad knowledge, taste, and agency are exactly what let people get outsized results from AI. The experts who gain most are the ones who know what to ask for and can tell when the answer is wrong How do human strengths help people exploit AI capabilities?.
Finally, the role is moving from doing the work to designing where humans step in. Nobody has solved when an agent should hand off to a person. Microsoft's Magentic-UI works around this by spreading human involvement across several points: planning together, splitting tasks, guarding risky actions, and verifying results When should human-agent systems ask for human help?. As agents start buying, deploying, and transacting, the bottleneck moves from capability to accountability: identity, delegation, and audit trails Does agent capability matter more than coordination infrastructure?. The Future of Life Institute goes a step further and argues that oversight can't be left to companies at all. In their view it needs binding government limits Can companies alone manage the risks of AI systems?. Taken together, the corpus suggests experts matter less as people who know more facts than AI. They matter as people who can be held accountable for a judgment, who can show the reasoning behind it, and whose peers can check it.
Sources 12 notes
METR's RE-Bench found AI agents score 4× higher than expert humans at 2-hour budgets but humans narrowly exceed agents at 8 hours and lead 2× at 32 hours, suggesting agents hit scaling plateaus while humans improve with extended effort.
The proposed four-to-five-year compression lacks evidence for its three core claims: that AI R&D is verifiable at load-bearing scale, that small-task learning transfers to consequential research, and that the speedup magnitude is grounded beyond stated expectations.
Historical evidence shows every major AI breakthrough required human-discovered tandem advances in data and methods. Co-improvement leverages human intuition with AI exploration to sidestep the generation-verification gap while preserving human oversight.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Expertise requires anticipating audience acceptability and social validity, not just retrieving information. AI lacks the mechanism to perform this communicative work, making its fluent output epistemically misleading despite its confident form.
Show all 12 sources
Research argues AI should supplement rather than replace human reasoning, with deference withdrawn when domain mismatch, bias, conflicting authority, or new evidence emerges. This prevents opacity-driven failures that full preemption would mask.
Experts are being repositioned to validate and manage AI outputs rather than produce original thinking. This custodial shift removes the labor of argumentation and testing that kept experts aligned with genuine knowledge production.
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
Mollick demonstrates that deep knowledge, wide knowledge, taste, and agency—paired with AI tools—enable accomplishments that would take weeks of traditional work. Experts who know their domain and what to ask AI systems see outsized returns on their efforts.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
Once agents move beyond simple API calls to purchasing, deploying, and transacting with real consequences, the bottleneck shifts from model capability to whether they can coordinate reliably, maintain accountability, and produce auditable evidence. Infrastructure—identity, delegation, attestation, and audit trails—matters more than marginal improvements to reasoning.
The Future of Life Institute argues that escalating AI incidents demonstrate private companies cannot self-police effectively, and calls for government-mandated limits on recursive self-improvement practices until safety research is complete, backed by hardware verification technology.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
- Epistemic Deference to AI
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery
- GenAI as a Power Persuader: How Professionals Get Persuasion Bombed When They Attempt to Validate LLMs
- Atria Dawn: The Dawn of Agentic Superintelligence
- A Rational Analysis of the Effects of Sycophantic AI
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- ASI-Evolve: AI Accelerates AI