Can small language models handle most agent tasks?

The dominance of LLMs in agentic AI design is both excessive and misaligned with functional demands. The majority of agentic subtasks in deployed systems are repetitive, scoped, and non-conversational — calling for models that are efficient, predictable, and inexpensive, not models with impressive generality and conversational fluency.

Three arguments support the position:

V1: SLMs are sufficiently powerful. Current SLMs handle the specific, well-defined language modeling tasks that constitute most agent invocations. The $5.6bn LLM API market sits beneath $57bn in infrastructure investment — a 10-fold discrepancy that assumes LLMs remain the cornerstone without substantial alteration.

V2: SLMs are more operationally suitable. Serving a 7B SLM is 10-30× cheaper than a 70-175B LLM in latency, energy, and FLOPs. Fine-tuning requires only GPU-hours not GPU-weeks. Edge deployment is feasible on consumer hardware. And SLMs may be more parameter-efficient: LLMs exhibit sparse activation patterns where most parameters don't contribute to any single output, while this behavior is more subdued in SLMs.

V3: SLMs are necessarily more economical. Per inference, per fine-tuning cycle, per deployment. The compounding effect across millions of agent invocations is enormous.

The architectural conclusion is heterogeneous agentic systems: SLMs handle all routine subtasks by default, LLMs are invoked selectively and sparingly for open-domain dialogue or general reasoning. This "Lego-like" composition — scaling out by adding small specialized experts instead of scaling up monolithic models — yields systems that are cheaper, faster to debug, easier to deploy, and better aligned with the diversity of real-world agent tasks.

Since Does model access level determine which specialization techniques work?, heterogeneous architectures multiply the relevance of this taxonomy — different agents in the same system may operate at different access levels. And since How do knowledge injection methods trade off flexibility and cost?, SLMs shift the Pareto frontier: fine-tuning is cheap enough that injection methods previously reserved for production-critical models become routine.

Routing as the enabling mechanism (from Arxiv/Routers): The SLM-first thesis requires a concrete mechanism for deciding when to escalate from SLM to LLM. The routing literature provides it. RouteLLM trains routers on preference data to predict when a weaker model suffices, achieving 40-50% cost reduction. Hybrid-LLM adds a tunable quality threshold adjustable at test time — exactly the knob a heterogeneous system needs to trade quality for cost per scenario. Avengers-Pro goes further: ten ~7B models with routing surpassed GPT-4.1 and 4.5, demonstrating that a pool of small models with good routing can outperform a single large one. This validates the SLM-first architecture empirically: the routing layer is not just a cost optimization but a performance optimization. See Can routers select the right model before generation happens? and Can routing beat building one better model?.

Source: Agents Multi Architecture