Can API-first agents outperform UI-based agent interaction?
This explores whether directing agents to use APIs instead of navigating UIs reduces task completion time and errors. The question matters because current LLM agents struggle with sequential UI steps that multiply latency and hallucination risk.
Current LLM-based UI agents suffer from two compounding problems: latency scales with sequential interactions (each UI step requires an LLM call with large visual context), and hallucination risk increases per step (each reasoning step adds probability of selecting a wrong UI control). Inserting a 2x2 table in Word requires "Insert → Table → 2x2 Table" — three sequential UI interactions, each requiring full UI state processing.
AXIS (Agent eXploring API for Skill integration) demonstrates that prioritizing API calls over UI interactions resolves both problems simultaneously:
- 65-70% task completion time reduction — API calls execute directly without sequential UI navigation
- 97-98% accuracy maintained — comparable to human performance
- 38-53% cognitive workload reduction — users specify intent, not procedures
The HACI (Human-Agent-Computer Interaction) paradigm shift: API-first agents replace UI agents, falling back to UI interaction only when relevant APIs are unavailable. API calls require fewer tokens and produce more reliable code-formatted responses compared to UI state descriptions.
The self-exploration mechanism is key to practicality: AXIS automatically explores existing applications, learns from support documents and action trajectories, and constructs new APIs from existing ones. This addresses the bootstrapping problem — APIs don't need to be manually created for every application.
Because Are reasoning model collapses really failures of reasoning?, the UI-to-API shift removes execution failure as a bottleneck. UI interaction is execution; API interaction is closer to specification. The agentic hierarchy becomes: user intent → agent reasoning → API execution, removing the fragile UI navigation layer.
This connects to Can reasoning and tool execution be truly decoupled?: API-first interaction is a structural form of the same decoupling — separating what the agent wants to do from the mechanics of how the application implements it.
Inquiring lines that read this note 37
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI agents improve their skills through accumulated experience and reuse? Should GUI agents use structured screen representations instead of end-to-end vision?- Why does explicit screen parsing outperform pure vision in GUI agents?
- Can API-first interaction replace traditional UI-based agent interfaces?
- What types of tasks benefit most from dynamically generated interfaces?
- How does API-first interaction compare to generative interface approaches?
- Why do static screenshot models fail to capture multi-step UI task intent?
- Can specialized perception components replace end-to-end vision in GUI agents?
- What makes accessibility trees insufficient compared to visual GUI understanding?
- Should GUI agents use intermediate structured representations instead of raw pixels?
- Why do APIs outperform UIs for agent task completion?
- Can screen perception be effectively decoupled from planning in GUI agents?
- Why do GUI agents need pixels while document systems can use bounding boxes?
- Should agents use APIs or GUI interaction for efficiency?
- Do dynamically generated interfaces perform better than pre-built ones?
- Should AI interfaces keep manual GUI controls as a fallback?
- Why do dynamic UIs reduce cognitive load but complicate user control and predictability?
- How does the execution layer constrain agent performance in tool use?
- How do agents discover and construct new APIs from existing applications?
- Why do 85 percent of production agents avoid third-party frameworks?
- Why do production AI agents deliberately stay simple and avoid frameworks?
- Can heterogeneous AI agents integrate through shared API and MCP interfaces?
- Do agent-created languages improve or degrade performance on their original tasks?
- What are the key interaction mechanisms that make human-agent collaboration work?
- Which workplace tasks remain hardest for AI agents to complete autonomously?
- Do autonomous workplace agents face different bottlenecks than consultation assistants?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Are reasoning model collapses really failures of reasoning?
Explores whether language models hit a fundamental reasoning ceiling or whether text-only evaluation masks execution limitations. Examines how tool access might reveal hidden reasoning capabilities.
API-first removes the execution failure layer
-
Can reasoning and tool execution be truly decoupled?
Can LLM reasoning be separated from tool observations to eliminate redundant re-prompting and enable parallel execution? Two recent architectures suggest yes, but what are the tradeoffs?
same decoupling principle applied to agent-application interaction
-
Why can't advanced AI models take initiative in conversation?
Despite extraordinary capability in answering and reasoning, LLMs fundamentally cannot initiate, redirect, or guide exchanges. Understanding this gap—and whether it's fixable—matters for building AI that truly collaborates rather than merely responds.
AXIS addresses execution-layer passivity: UI-based agents passively follow sequential interaction steps determined by the application, while API-first agents directly specify intent; the HACI paradigm shift parallels the conversational passivity diagnosis — both reveal that current agent architectures are reactive to their environment (UI state, user query) rather than proactive
-
Why do AI agents fail at workplace social interaction?
Explores why current AI agents struggle most with communicating and coordinating with colleagues in realistic workplace settings, despite strong reasoning capabilities in other domains.
AXIS directly addresses one of the two hardest failure modes (complex UI navigation), potentially raising the 30% ceiling for workplace tasks that require professional tool interaction
-
Do generated interfaces outperform text-based chat for most tasks?
Explores whether LLMs should create interactive UIs instead of text responses, and under what conditions users prefer dynamic interfaces to traditional conversational chat.
converging evidence from opposite directions: AXIS moves agents from UI-based to API-based interaction (65-70% faster); generative interfaces move users from chat to dynamically generated UIs (70% preference); both challenge the chat paradigm as default and point toward intent-specification over procedure-following as the interaction model
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
- Delegating or Doing? Understanding User Behavior in Hybrid Human-Agent Interfaces
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
- Towards a Science of Scaling Agent Systems
- Toward Efficient Agents: A Survey of Memory, Tool Learning, and Planning
- Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- How AI Impacts Skill Formation
Original note title
API-first agent interaction reduces task completion time by 65 to 70 percent compared to UI-based agent interaction — reframing human-agent-computer interaction