Domain Specialization in LLMs

Can conversational AI safely take patient histories without supervision?

A feasibility study tested whether an LLM-based system could conduct real clinical interviews with urgent-care patients without requiring safety interventions. Understanding AI safety in unsupervised clinical settings matters for potential deployment.

Can machines learn to predict which research ideas will work?

Can a fine-tuned language model with access to published papers predict which unimplemented AI ideas will succeed empirically, and would it outperform human researchers making the same judgment?

Do AI coding tools actually speed up experienced developers?

Developers predicted AI tools would make them 24% faster, but a randomized trial measuring real work found the opposite. Understanding this gap between forecast and outcome matters for assessing AI's real productivity impact.

Did conference reviewers prefer AI reviews over human ones?

AAAI-26 ran a live pilot adding labeled AI reviews to 22,977 papers alongside human reviews. A survey asked participants which reviews were more useful, particularly on technical accuracy and research suggestions.

Can micro-frictions boost rivalry without harming collaboration?

The survey proposes adding small frictions to GenAI writing tools to encourage a rivalrous stance while preserving collaborative benefits. No intervention has tested whether this design actually works or what unintended effects might emerge.

Can language models reason better than physicians at diagnosis?

An LLM was tested on challenging clinical cases against hundreds of physicians as a baseline. The research asks whether AI can match or exceed human diagnostic reasoning in structured medical settings.

What stops AI from discovering science without human help?

Can current agentic AI systems autonomously conduct natural-science discovery, or do fundamental gaps in training and deployment block them? This matters because it shapes realistic expectations for AI in research.

Can AI agents produce scientifically novel and important ideas?

A 2025 conference let AI agents lead research and peer review, accepting 48 of 314 papers. Reviewers found technically sound work but questioned whether it addressed questions that actually matter to science.

Do accepted papers need more human guidance than rejected ones?

Agents4Science organizers reported that accepted papers involved more human input than rejected papers, with humans leading design and AI handling analysis. This raises whether human guidance predicts acceptance and how labor should divide in AI-authored research.

Does AI assistance help less experienced workers most?

When customer support agents gain access to an AI chat assistant, do productivity gains concentrate among newer, less skilled workers? Understanding this pattern matters for knowing who benefits from AI tools and whether deployment widens or narrows workplace skill gaps.

Do AI coding features actually speed up engineer productivity?

A randomized trial of Google engineers tested whether AI-powered coding tools reduce time spent on complex tasks. Understanding real-world productivity gains matters as companies invest heavily in these features.

Can two-stage review and badges fix AI conference peer review?

A position paper diagnoses AI conference review failures across authors, reviewers, and venues, proposing staged author feedback on reviews and a reviewer reward system. Does this approach actually reduce bias and improve review quality?

Why don't radiologists benefit from AI predictions?

When radiologists receive AI predictions, they often fail to incorporate them properly into their decisions. This explores what belief-updating errors prevent radiologists from realizing potential AI-assisted gains.

Can AI systems safely replace human peer reviewers?

Explores whether AI reviewers meet two critical conditions for automation: maintaining diverse perspectives and resisting score manipulation. Tests whether current systems are ready to handle peer review at scale.

Can AI systems generate research papers that pass peer review?

Whether fully autonomous AI can produce manuscripts meeting publication standards in real peer-review settings. This tests whether current scientific gatekeeping processes can already validate AI-generated research.

Does AI help individual scientists while narrowing scientific focus?

An analysis of 41 million papers explores whether AI adoption simultaneously boosts individual researcher productivity and citations while constraining the breadth of topics science collectively investigates.

Can an AI system diagnose better than primary care doctors?

A study compared AMIE, an LLM trained through self-play simulation, against 20 primary care physicians on 149 clinical cases evaluated by specialists. The question asks whether AI can genuinely outperform human doctors in diagnostic reasoning, and what that means for clinical practice.

Can AI systems generate hypotheses that match unpublished experimental discoveries?

Researchers tested whether an AI hypothesis-generation platform could arrive at mechanisms their own labs had experimentally confirmed but not yet published, exploring whether AI can independently discover known-but-hidden biological answers.

Does AI assistance erode the skills needed to oversee it?

Anthropic engineers report productivity gains from Claude but worry that heavy delegation may wear down the coding skills required to validate its work. The tension raises questions about whether AI collaboration trades expertise for output.

How are national lab staff actually using generative AI?

This research explores whether generative AI adoption at a US national lab has moved beyond experimentation into routine work. Understanding real usage patterns helps clarify what AI is genuinely changing about knowledge work.

Can peer review gates stop the flood of AI-generated surveys?

arXiv CS now requires prior journal or conference peer review for survey and position papers. The question is whether this upstream gate actually reduces low-quality submissions and restores the moderation workload to manageable levels.

Can authors rank their own papers better than peer reviewers?

Do researchers have better insight into which of their own submissions will prove influential than official peer reviewers do? This matters because peer review is expensive and may miss work with long-term scientific value.

Does automation raise or lower the skills that remaining work demands?

When automation removes tasks from a job, does it make the leftover work require more expertise or less? This matters because it determines whether workers earn more or fewer opportunities in that occupation.

Why do autonomous research systems release code but not verification artifacts?

Autonomous research systems publish their code at high rates, yet rarely share the seeds, traces, or novelty checks that would let reviewers verify their claims. What explains this gap and what would close it?

How much peer review text shows signs of LLM modification?

Researchers analyzed AI conference reviews to estimate what fraction might have been substantially altered by large language models. Understanding this helps clarify how AI tools are entering academic peer review.

Is generative AI displacing workers at economy-wide scale?

Researchers examine whether AI has caused broad job losses across the U.S. economy using detailed payroll records. Understanding displacement patterns matters for policy and worker planning.

Can clinicians tell GPT-4 advice apart from expert advice?

This study explores whether trained clinicians can distinguish AI-generated psychological advice from expert advice, and how they rate the quality and empathy of each. The question matters for understanding whether AI might reliably supplement human expertise in mental health settings.

Does more thinking time improve AI-generated research hypotheses?

Co-Scientist's builders tested whether giving a multi-agent AI system more compute during hypothesis generation produces better scientific ideas. They measured Elo ratings across 203 research goals to explore this scaling relationship.

Does personal preference shape how engineers use AI tools?

This study explores whether engineers choose their own level of AI reliance or whether company policies decide it for them. The question matters because it determines where control over AI systems actually lies in software teams.

Why do language models fail at temporal reasoning in complex tasks?

Language models correctly answer simple temporal questions but produce logically impossible timelines in complex legal documents. This explores what task features trigger reasoning failures and whether the competence is genuinely lost or masked by surface-level patterns.

Does AI chatbot adoption change worker pay and hours?

Two years after ChatGPT's release, did widespread adoption of AI chatbots by Danish employers shift worker earnings or time on the job? Understanding timing matters for predicting when AI's labor market effects become visible.

Can we measure whether AI erodes independent skill?

Current telemetry tracks how people use AI but not whether they become more capable without it. Existing measurement tools cannot yet determine if AI helps or hurts skill formation at scale.

Does medical AI need knowledge or reasoning more?

Medical and mathematical domains may require fundamentally different AI training priorities. If medical accuracy depends primarily on factual knowledge while math depends on reasoning quality, should we build and evaluate these systems differently?

Does model access level determine which specialization techniques work?

Different specialization approaches require different levels of access to a model's internals. Understanding this constraint helps practitioners choose realistic techniques for their domain adaptation goals.

How much does wrong AI advice harm radiologist accuracy?

When mammography radiologists receive incorrect AI suggestions labeled as system output, how much does their diagnostic accuracy decline? This matters for understanding automation bias in clinical workflows.

Can automation raise output while slowing growth?

Entry-level automation can boost immediate productivity but reduce long-term growth if it disrupts how novices learn from top experts. The question asks whether employment headcounts alone miss what matters for welfare.

Does time pressure make AI advice more persuasive to experts?

When pathologists work under time constraints, does pressure to decide quickly make them more likely to trust and act on AI recommendations, even when those recommendations are wrong?

How much scientific writing has LLMs actually modified?

Researchers estimated the fraction of LLM-modified content across nearly one million scientific papers from 2020 to 2024. Understanding these population-level trends matters for gauging AI's real impact on scientific publishing.

Does AI-enriched analyst reports improve forecast accuracy?

When generative AI is integrated into analyst platforms, does the richer information and broader coverage it produces translate into better forecasts? This matters because output quality and decision accuracy may diverge.

Can LLM feedback help peer reviewers improve their own reviews?

This randomized trial tested whether optional AI-generated suggestions on review quality would prompt reviewers to revise, and whether those revisions would be more useful to authors and decision-makers.

Can institutional publication records train better scientific evaluators?

Can AI models learn to make reliable low-verifiability judgments by training on where and what fields published, rather than explicit quality rubrics? This matters because science depends on gatekeeping decisions that individual reviewers struggle to make consistently.

Do social networks drive adoption of new coding tools?

This research explores how Microsoft engineers first encountered and started using agentic CLI coding tools, and whether peer influence through social networks shaped early adoption patterns.

Did ChatGPT's release reduce freelance writing work and pay?

Did the introduction of generative AI in late 2022 cause measurable drops in employment and earnings for freelancers in occupations most exposed to the technology, particularly writing roles on online labor platforms?

Can large language models reliably find software vulnerabilities?

Frontier LLMs claim to detect security flaws but may flag false positives at high rates and miss real vulnerabilities. Understanding whether they can be trusted for cybersecurity work matters as organizations consider deploying them.

How widely do peer reviewers actually use AI tools?

A survey of 1,645 active researchers reports that 53% of peer reviewers now use AI in review, with adoption highest among early-career researchers. The finding raises questions about whether self-reported usage reflects actual practice and whether policy should follow or lead this trend.

How close are frontier AI models to expert work quality?

GDPval benchmarked frontier models on 1,320 expert-built tasks across 44 occupations, using head-to-head expert judgment to measure whether AI is approaching human deliverable quality in knowledge work.

Why doesn't mathematical reasoning transfer to medicine?

Can models trained to reason well about math apply those skills to medical domains through fine-tuning? This explores whether reasoning ability is truly domain-agnostic or constrained by domain-specific knowledge requirements.

Can generative AI replace the benefits of having a human teammate?

This experiment tested whether professionals working alone with AI could match the performance of teams working without AI on real product innovation tasks. Understanding this matters for how organizations might restructure work around AI tools.

Can expert-written questions resist web-assisted non-expert answering?

GPQA tested whether graduate-level multiple-choice questions written by domain experts remain difficult even when non-experts have unrestricted internet access. This matters for building benchmarks that can supervise AI systems on tasks beyond typical human reach.

Can GPT-4 feedback match what human reviewers catch?

Does an LLM reviewing system raise the same points that human reviewers do? This matters because it tests whether AI could complement or replace human peer review for research papers.

Does GPT-4 actually improve the quality of legal analysis?

A randomized trial of law students using GPT-4 for legal tasks found large speed gains but small quality improvements. The finding raises questions about whether AI assistance genuinely enhances analytical capability or mainly accelerates output.

How many accepted conference papers contain hallucinated citations?

A vendor scan of NeurIPS 2025 papers flagged hundreds of citations that could not be verified online. The question is whether these flags represent genuine hallucinations or unverifiable but real sources, and how many require correction.

When do graph databases outperform vector embeddings for retrieval?

Vector similarity struggles with aggregate and relational queries that require traversing multiple entity connections. Can graph-oriented databases with deterministic queries solve this failure mode in enterprise domain applications?

How much GPT-written scholarship reaches Google Scholar undetected?

Haider et al. searched Google Scholar for telltale ChatGPT phrases to estimate how many papers contain undisclosed AI authorship, especially in policy-relevant fields. Understanding prevalence matters because lay readers—politicians, patients, students—may treat these papers as credible research.

Are hidden AI prompts in preprints a deceptive research practice?

Eighteen arXiv papers contained white-text instructions targeting AI reviewers with favorable-review requests. The question is whether this represents a novel form of research misconduct or something else entirely.

Does balancing rivalry and collaboration with GenAI boost writer productivity?

Professional writers report different work practices depending on whether they view GenAI as a rival or collaborator. This explores whether combining both orientations produces stronger outcomes than holding either alone.

Does a strong track record protect freelancers from AI?

When ChatGPT launched, did established freelancers with high ratings and employment history face less disruption than newer workers? The answer reveals whether reputation acts as a buffer against technological displacement.

How can conferences detect and handle LLM misuse in peer review?

Explores how ICLR 2026 balanced detection limitations with practical enforcement, distinguishing between acceptable LLM assistance and problematic offloading of reviewing responsibilities.

How do detection tools shape LLM use enforcement at ICLR?

ICLR's 2026 policy uses LLM detectors to flag papers, but requires human reviewers to find concrete evidence before acting. This matters because false positives from automated tools could harm unflagged papers while creating extra work for area chairs.

How many peer reviewers secretly used LLMs despite the ban?

ICML used hidden watermarks in submission PDFs to detect LLM-written reviews submitted under a no-LLM policy. The question explores whether 795 flagged reviews represent the true scale of LLM use or only careless violations.

Will AI automation widen science's productivity versus progress gap?

As AI makes it easier to publish more papers, will it trap scientists in chasing metrics rather than breakthroughs? The concern is that automation amplifies existing incentives that reward output over discovery.

How do knowledge injection methods trade off flexibility and cost?

When and how should domain knowledge enter an AI system? This explores the speed, training cost, and adaptability trade-offs across four injection paradigms, and when each approach suits different deployment constraints.

Can a shared world model sustain coherence across hundreds of agent steps?

Kosmos claims a structured world model coordinates data and literature agents over 12-hour runs with 79.4% accuracy. The question is whether this mechanism actually causes the coherence, or whether other factors explain the results.

How often do legal AI tools actually hallucinate citations?

Legal vendors claim their AI research tools eliminate hallucinations, but do they? This preregistered study measures hallucination rates in leading commercial legal-research systems to test those marketing claims.

Do LLM improvements reflect reasoning gains or corpus shifts?

When large language models improve on previously failed tasks, does this show they've learned to reason better, or are they simply reflecting changes in human-written text they train on? Understanding this matters for assessing what LLMs actually know.

Does LLM assistance help clinicians build better differentials?

A randomized study tested whether giving clinicians access to an LLM improved their diagnostic reasoning on challenging cases. Understanding this matters for evaluating AI's role in clinical decision support beyond standalone performance.

Can AI safety nets reduce errors in live clinical practice?

A study of 39,849 visits at Nairobi primary care clinics tested whether LLM decision support tools could help clinicians make fewer diagnostic and treatment errors in routine care, and what conditions made the tool effective.

Does LLM writing assistance change how scientists publish?

When scientists adopt LLMs to draft manuscripts, do they produce more papers, and does writing quality still signal merit? This matters because it affects how we evaluate scientific work.

Why do LLMs fail when users interact with them?

Standard benchmarks show LLMs excel at medical diagnosis alone, yet real users get no benefit. This explores where the breakdown happens between model capability and human decision-making.

Do language models possess tacit knowledge in Davies' sense?

Explores whether transformer LLMs meet philosophical criteria for tacit knowledge—rules that causally guide behavior without explicit storage. The question matters because it reframes how we understand what LLMs learn and how to intervene in their representations.

Why do language models struggle with historical legal cases?

Explores whether LLMs' training data recency bias creates systematic performance degradation on older cases, and what this reveals about how models represent temporal information in specialized domains.

Do benchmark gains in medical AI reflect real-world progress?

Med-Gemini achieves 91.1% on MedQA, but clinician review found ~7% of questions have missing information or labeling errors. The question is whether such benchmark improvements actually signal meaningful advances in clinical capability.

Does medical model architecture or training data drive performance gains?

MedGemma claims its medical improvements come from domain-specific training data rather than architectural changes. But the evidence comes only from the developers' own benchmarks, without independent validation or real-world clinical testing.

Does medical pretraining actually improve model performance?

When medical models are fairly compared to their base counterparts with per-model prompt tuning and statistical significance testing, do they consistently outperform on medical question answering tasks?

Did developers opt out of METR's AI study because of selection bias?

METR's August 2025 developer productivity study may have missed its most AI-dependent workers. Understanding whether selection effects—developers refusing to work without AI tools—distorted the speedup estimates matters for interpreting what the data actually shows about AI's real-world impact.

Can unreviewed preprints shape scientific debate before peer review?

Explores whether papers posted to preprint servers influence research discussions and policy decisions despite lacking peer review, and what happens when institutions later challenge their reliability.

Do LLM benchmarks actually measure what they claim to measure?

A systematic review examined whether 445 LLM benchmarks have sound construct validity—whether their tasks and metrics truly capture the phenomena they're designed to test. This matters because flawed benchmarks can mask model failures and mislead research.

Does NeurIPS 2025's LLM disclosure policy match what readers actually need?

NeurIPS 2025 requires LLM disclosure only for methodological use, not writing help. But reader studies suggest disclosure matters more broadly, raising questions about whether the policy's limits adequately protect scientific integrity.

Why did single-factor NHANES studies explode after 2021?

NHANES papers proposing one-predictor associations surged from 4 per year to 190 in 2024, raising questions about what enabled the volume spike and whether design shortcuts became systematically common.

Are researchers hiding prompt injections in academic papers?

Nikkei Asia discovered hidden text in research papers from multiple institutions instructing AI summarizers to generate positive reviews. This raises questions about whether such injections successfully manipulate AI evaluation and how widespread the practice is.

Did GPT-5 really solve previously unsolved math problems?

OpenAI claimed GPT-5 solved hard Erdős problems open for decades. But what did the model actually do, and how was the claim verified or challenged by domain experts?

Can orchestration strategies boost diagnostic AI without better models?

This research explores whether structuring how models collaborate—through virtual panels, cost estimation, and ensembling—can improve medical diagnosis accuracy and efficiency beyond what individual models achieve alone.

Why do specialized models fail outside their domain?

Deep domain optimization creates sharp performance cliffs at domain boundaries. Specialized models generate plausible-sounding but ungrounded responses when queries fall outside their training scope, and often fail to signal their own ignorance.

How much AI content appears in peer review at ICLR?

A detection company scanned 70,000 ICLR reviews to estimate how prevalent AI-generated text is in peer review. Understanding this prevalence matters for assessing review quality and integrity in academic publishing.

Does peer review quality collapse under submission overload?

As submissions rise, do overtaxed reviewers become less accurate, and does this lower quality trigger more speculative submissions in a reinforcing cycle? Understanding this mechanism matters for diagnosing peer review's capacity crisis.

Does the label on advice shape how clinicians judge it?

When clinicians believe advice comes from an expert, do they rate it higher regardless of who actually wrote it? This matters because it reveals whether judgments track the advice itself or just its claimed source.

Do small law firms misuse AI more often than large ones?

A database of 114 court cases with AI-tainted filings shows 90 percent involved small or solo firms. But does this reflect higher misuse rates, or simply better detection of errors in smaller practices?

Can prompt optimization teach models knowledge they lack?

Explores whether sophisticated prompting techniques can inject new domain knowledge into language models, or if they're limited to activating existing training knowledge.

Does labeling advice as AI change how clinicians use it?

When physicians know diagnostic advice comes from an AI system rather than a human expert, do they rely on it differently? This matters because AI labels might trigger skepticism that affects clinical decisions.

Where is most LLM-generated content actually appearing in computer science?

Review papers show higher LLM-generated shares than other papers, but the raw volume tells a different story. This note explores which paper types actually contain most generated content by count.

Can simple rewards alone teach complex domain reasoning?

Does reinforcement learning on difficult problems with basic accuracy rewards produce sophisticated reasoning strategies without explicit chain-of-thought training? This challenges assumptions about what domain AI models need to learn effectively.

Does RL improve domain reasoning by adding knowledge or removing it?

When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.

Can multi-agent systems guide wet-lab discovery through iterative cycles?

Robin pairs literature-mining agents with a data-analysis agent to propose disease targets and drug candidates, cycling bench results back into hypothesis generation. The question explores whether this loop produces valid discoveries and what role human judgment should play.

Can AI-generated papers pass peer review undetected?

Explores whether end-to-end AI-generated manuscripts can clear human double-blind review at academic workshops, and what acceptance rates reveal about reviewer capability to distinguish AI from human work.

Does scientific fraud operate through organized networks or individual actors?

Richardson et al. investigate whether fraudulent publishing emerges from coordinated systems of paper mills, brokers, and editors, or from isolated bad actors. Understanding the structure matters for designing detection and prevention strategies.

Does supervised fine-tuning actually improve reasoning quality?

While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.

Do LLM reviewers favor papers written by other LLMs?

When LLMs serve as peer reviewers, do they systematically score papers differently based on whether they were written by humans or LLMs? Understanding this matters for fair and trustworthy publication systems.

Does AI polyp detection weaken endoscopists' unassisted performance?

When endoscopy centers introduce AI-assisted polyp detection, do clinicians' skills deteriorate when working without the tool? This matters because overreliance could erode diagnostic ability even as AI improves overall detection.

Can organizing knowledge structures beat raw training data volume?

Does structuring domain knowledge into taxonomies during training enable models to learn more efficiently than simply increasing the amount of training data? This challenges assumptions about scaling knowledge injection.

Does AI create a coupled arms race in research production and review?

How do AI-driven changes to research production and peer review interact as a single feedback system? Understanding this coupling matters for designing sustainable evaluation mechanisms that remain trustworthy at scale.

Can one AI system complete a full research cycle end-to-end?

This explores whether a single agentic system can autonomously handle ideation, coding, experiments, writing, and peer review—and whether outputs from such a system can pass human evaluation at research venues.

Do LLM reviewers actually favor LLM-written papers?

Does the apparent bias of LLM-assisted peer reviewers toward LLM-generated papers reflect genuine preferential treatment or an artifact of quality distribution? The answer shapes how we interpret reviewer behavior.

Does supervised fine-tuning improve reasoning or just answers?

Explores whether training models on question-answer pairs actually strengthens their reasoning quality or merely optimizes them toward correct outputs through shortcuts. This matters for deploying AI in domains like medicine where reasoning must be auditable.

Do junior developers choose AI based on their ability to verify results?

Can junior developers reliably decide when to use AI by assessing whether they can check the output themselves? This matters because it reveals how newcomers self-regulate AI use amid pressure to adopt it quickly.

Did automated text tools produce suspicious phrases in one journal?

A 2021 analysis of a major computer science journal found concentrated clusters of odd phrases alongside shortened review timelines, raising questions about whether AI text generation tools may have contributed to questionable publications.

Can asynchronous expert training beat synchronized distributed LLM training?

Can training domain-specialized LLM copies in parallel without synchronization, then merging their components into a routed mixture, achieve better efficiency and accuracy than keeping all copies synchronized?