A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy

Recent improvements in large language models (LLMs) have led many researchers to focus on building fully autonomous AI agents. This position paper questions whether this approach is the right path forward, as these autonomous systems still have problems with reliability, transparency, and understanding the actual requirements of human. We suggest a different approach: LLM-based Human-Agent Systems (LLM-HAS), where AI works with humans rather than replacing them. By keeping human involved to provide guidance, answer questions, and maintain control, these systems can be more trustworthy and adaptable. Looking at examples from healthcare, finance, and software development, we show how human-AI teamwork can handle complex tasks better than AI working alone. We also discuss the challenges of building these collaborative systems and offer practical solutions. This paper argues that progress in AI should not be measured by how independent systems become, but by how well they can work with humans.

This paper advocates a fundamental shift from pursuing autonomous LLM agents to prioritizing collaborative HAS. We begin by critically examining the current trajectory towards fully autonomous LLM-based agents, identifying limitations in reliability, complexity handling, and ethical concerns (Section 2). Building on these identified shortcomings, we present LLM-based HAS as a paradigmatic alternative, establishing foundational principles that demonstrate how human-LLM collaboration directly addresses the core weaknesses of fully autonomous approaches (Section 3). To substantiate our position, we highlight the emerging significance of HAS across multiple domains, showcasing promising results (Section 4). We acknowledge that HAS approaches face their own challenges; Therefore, we list these key limitations and propose concrete research directions to address these issues (Sections 5 & 6). Finally, we present Alternative Views in Section 7.

3.1 LLM-based Human-Agent Systems

Definition 3.1 (LLM-based Human-Agent Systems). An LLM-based Human-Agent System is a collaborative framework where humans and LLM-powered agents interact to accomplish tasks. Unlike fully autonomous agents, these systems maintain humans in the loop to provide critical information and clarifications [44, 62, 108], offer feedback by evaluating outputs and guiding adjustments [30, 24, 52], and assume control in high-stakes or sensitive scenarios [12, 63, 93]. This human involvement in LLM-HAS ensures enhanced performance, reliability, safety, and explicit accountability, particularly where human judgment remains indispensable.

3.2 Advantages of LLM-HAS

The advantages and rationale for prioritizing HAS stems directly from its potential to address the critical limitations and risks associated with autonomous agent systems:

Improved Trust and Reliability: The interactive nature of HAS allows humans to provide crucial feedback, correct potential LLM hallucinations in real-time [99], verify information, and guide the agent toward more accurate and reliable outputs [89]. This collaborative verification process is essential for building trust, especially where the cost of error is high.

Managing Complexity and Ambiguity. Unlike autonomous agents that struggle with unclear instructions, LLM-HAS excels through continuous human clarification [31]. Humans provide essential context, domain expertise, and progressive refinement of ambiguous goals—critical capabilities for complex tasks. When faced with an underspecified objective, the system can request clarification rather than proceeding with potentially incorrect assumptions [60], making LLM-HAS particularly effective for open-ended research or creative endeavors where objectives evolve dynamically [22]. Clearer Lines of Accountability: With a human involved in the decision-making process, especially in supervisory or interventional roles, establishing accountability becomes more straightforward. The human operator or supervisor can often be designated the responsible party, simplifying the legal and regulatory landscape compared to situations where an autonomous agent makes a critical error [97].

[Model Engineering] Lacking Adaptivity and Continuous Improvement. A core challenge in LLM-HAS development is building truly adaptive and continuously improving AI teammates. Previous approaches treat LLMs as fixed, pre-trained tools, thereby missing opportunities for dynamic evolution within collaborative settings [56]. This static view introduces three key challenges. First, most systems fail to adequately leverage human insights. Without advanced ways to incorporate diverse human guidance (e.g., preferences, critiques), LLMs struggle to become genuinely teachable and context-aware [98, 2]. Second, models lack robust capacity for continual learning and knowledge retention in dynamic environments. This prevents them from building long-term expertise and can lead to catastrophic forgetting, severely hindering their growth as collaborators [41, 50]. Third, the absence of real-time optimization—such as adaptive prompting and self-correction—hampers efficiency, alignment, and resource use [75, 81]. Effectively addressing these challenges through integrated human feedback, lifelong learning, and dynamic optimization is key to unlocking the full potential of LLM-HAS in human-agent collaboration.