Should human oversight precede fully autonomous AI agents?

The dominant research trajectory pursues fully autonomous LLM agents. This position paper argues the priority should be LLM-based Human-Agent Systems (LLM-HAS) — collaborative frameworks where humans remain in the loop to provide critical information, offer feedback, and assume control in high-stakes scenarios.

The argument rests on three structural advantages of collaboration over autonomy:

1. Improved trust and reliability — Interactive verification lets humans correct hallucinations in real-time and guide agents toward accurate outputs. This is essential where the cost of error is high.

2. Managing complexity and ambiguity — Autonomous agents struggle with unclear instructions. LLM-HAS enables continuous human clarification: providing context, domain expertise, and progressive refinement of ambiguous goals. The system can request clarification rather than proceeding with potentially incorrect assumptions.

3. Clearer accountability — With humans in supervisory or interventional roles, establishing accountability is straightforward. The human operator can be designated the responsible party, simplifying the legal and regulatory landscape.

However, the paper identifies three unsolved challenges for LLM-HAS itself:

• Leveraging human insights — Most systems fail to incorporate diverse human guidance (preferences, critiques) into model behavior, making LLMs not genuinely teachable.
• Continual learning — Models lack robust capacity for knowledge retention in dynamic environments, leading to catastrophic forgetting and hindering long-term collaborative growth.
• Real-time optimization — The absence of adaptive prompting and self-correction hampers efficiency and alignment.

This connects to When should human-agent systems ask for human help? — Magentic-UI operationalizes the HAS vision with concrete interaction mechanisms. It also extends Why do AI agents misalign with what users actually want? by arguing the fix is architectural (keep humans in the loop) not just capability-based (make models better at eliciting preferences).

The insight challenges the framing that AI progress = increasing independence. Instead: progress should be measured by how well systems work with humans, not how much they can do alone.