MasRouter: Learning to Route LLMs for Multi-Agent Systems

Multi-agent systems (MAS) powered by Large Language Models (LLMs) have been demonstrated to push the boundaries of LLM capabilities, yet they often incur significant costs and face challenges in dynamic LLM selection. Current LLM routing methods effectively reduce overhead in single-agent scenarios by customizing LLM selection for each query, but they overlook the critical decisions regarding collaboration modes and agent roles in MAS. In response to this challenge, we first introduce the problem of Multi-Agent System Routing (MASR), which integrates all components of MAS into a unified routing framework. Toward this goal, we propose MasRouter, the first high-performing, cost-effective, and inductive MASR solution. MasRouter employs collaboration mode determination, role allocation, and LLM routing through a cascaded controller network, progressively constructing a MAS that balances effectiveness and efficiency.

On the performance perspective, multi-agent systems have been proven to outperform single-agent approaches significantly (Chen et al., 2023b; Pina et al., 2023). On the cost perspective, though impressive in performance, existing multi-agent pipelines inherently introduce substantial token overhead and increased economic costs

For example, for software development tasks, an ideal MAS routing method could design a hierarchical workflow with sequential steps such as requirements analysis, algorithm design, code development, and testing, each requiring corresponding role profiles (Ramin et al., 2020; Zingg et al., 2023). Against this backdrop, we argue that routing in MAS involves more tasks than just LLM recommendations: ❶ Collaboration Mode Determination: Choosing the optimal communication mechanisms (e.g., Chain (Qian et al., 2023), Tree (Ishibashi and Nishimura, 2024), Graph (Hao et al., 2023)) for varying task complexities. This involves identifying the most efficient and adaptable multi-agent topology (Zhuge et al., 2024; Zhang et al., 2024a) that minimizes overhead while ensuring flexibility and scalability in more complex scenarios. ❷ Dynamic Agent Number: Determining the number of expert agents required (Huang et al., 2024; Aghdam et al., 2024) based on the difficulty of the input. ❸ Agent Role Allocation: Selecting suitable role to the agent according to the query domain (Chen et al., 2023b; Feng et al., 2024) to ensure efficient task division, creating a system greater than the sum of its parts (Shang et al., 2024). ❹Agent LLM Routing: Assigning each agent the appropriate LLM based on the collaborative topology and the role of each LLM (Feng et al., 2024).

In light of the scrutinizing challenges, we for the first time introduce the concept of LLM-based :: Multi-:: Agent :System:: Routing (MASR): Multi-Agent Systems Routing (MASR): Given a pool of available LLMs, collaborative communication modes, and possible agent roles, an optimal MAS Router for any query q should: (1) identify appropriate multi-agent collaboration modes, (2) allocate agent roles efficiently, and (3) assign the appropriate LLM to each agent, thereby balancing performance and cost. The formal definition of MASR is provided in Section 3. To construct a router that ideally adheres to the MASR principles, we propose an effective, token-economical, and inductive LLM-powered :: Multi- :Agent :System :::::: Router, termed MasRouter. Technically, MasRouter integrates collaboration mode determiner, agent role allocator, and agent LLM router into a unified routing framework: ❶ Collaboration determiner employs a variational latent variable model to route the user query to a suitable collaboration module; ❷ Role allocator progressively generates agent roles through a structured probabilistic cascade; ❸ LLM router models the LLM backbone recommendation for each agent as a multinomial distribution problem. Ultimately, MasRouter constructs a MAS that simultaneously balances effectiveness and efficiency.

Our contributions can be summarized as follows: • Problem Definition. We for the first time formally define Multi-Agent System Routing (MASR), which specifies the requirements for MAS routing: assigning the appropriate collaboration mode, agent roles, and LLMs to each query, thereby improving response quality and reducing unnecessary overhead.

• Practical Solution. We propose MasRouter, a modular MASR solution that utilizes a cascaded controller network to construct a highperforming and resource-efficient MAS progressively. Besides, MasRouter can seamlessly integrate with mainstream MAS to achieve efficient routing with significantly lower inference cost.

• Experimental Validation. Extensive experiments across five benchmarks show that MasRouter is: (I) high-performing, surpassing RouterDC, the state-of-the-art routing method, by 3.51% on average; (II) economical, reducing the overhead on HumanEval from 0.363$ to 0.185$; (III) inductive and plug-and-play, generalizing to unseen LLM backbones and collaboration modes and seamlessly combining with mainstream multiagent systems with 17% ∼ 28% ↓ fewer cost.