Can we automatically optimize both prompts and agent coordination?
This explores whether language agents can be represented as computational graphs whose structure and content adapt automatically. Why it matters: current agent systems require hand-engineered orchestration; automatic optimization could unlock more capable multi-agent systems.
The computational graph representation of language agents resolves a fragmentation problem: diverse prompting techniques (CoT, ToT, Reflexion, Self-Consistency) and multi-agent frameworks (AutoGPT, CAMEL, ChatDev) all look different in implementation but share the same underlying structure.
Three levels of the hierarchy:
- Node — a fundamental operation: LLM inference, tool use, function call, embodied action
- Graph (Agent) — multiple nodes forming a coherent functional entity; edges define execution topology
- Composite Graph (Swarm) — agents whose collective capabilities exceed individual agents; inter-agent edges establish collaboration
This is not merely descriptive. The graph representation enables automatic optimization along two axes:
- Node optimization — each node adapts its prompts based on previous input and task feedback (analogous to prompt tuning)
- Edge optimization — the connectivity between nodes changes, allowing suboptimal agent organization to be overcome and prompting techniques to be automatically recombined
Because Can reasoning topologies be formally classified as graph types?, the insight here is that the graph representation extends beyond individual reasoning traces to entire agent systems. A single agent's reasoning is a graph; a multi-agent system is a composite graph. The same formalism covers both, enabling optimization at both levels.
The Society of Mind (Minsky 1988) framing is deliberate: higher-level intelligence emerges from combining simpler, modular cognitive components. This is the same principle that makes Can extreme task decomposition enable reliable execution at million-step scale? work — decomposition into graph nodes enables both parallelism and optimization.
The practical implication: instead of hand-engineering agent orchestration patterns, define the problem as a graph and let optimization discover the topology. This is the agent-systems analog of what Do reasoning cycles in hidden states reveal aha moments? found for reasoning traces — topology matters, and it's optimizable.
Inquiring lines that read this note 43
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What causes coordination failures in multi-agent language model systems?- Can language agents be represented as optimizable computational graphs?
- How do graph-based reasoning topologies map to multi-agent interaction patterns?
- How do language agents implement prompts as executable computational graphs?
- How do language agents become optimizable computational graphs automatically?
- How do postmortem convention-setting stages enable language evolution in agents?
- When should you optimize agent behavior versus tool performance separately?
- How do agents discover and select which tools to invoke?
- Should agent capability be optimized separately from general capability?
- Can agents learn to use scaffolding structure the way they learn token weights?
- Why do rigid orchestration frameworks fail where generative environment specifications succeed?
- Should new agent protocols replace existing ones or layer on top of them?
- Can prompt engineering fully prevent role flipping in LLM agents?
- Can prompt optimization inject genuinely new knowledge into a model?
- Can prompt optimization for clarity automatically improve token efficiency?
- What knowledge can prompt optimization actually activate in trained models?
- How do external prompt artifacts improve agent behavior compared to inline instructions?
- Can adaptive prompt-difficulty allocation compound with architectural efficiency improvements?
- Can compute-optimal scaling work without co-optimizing the prompt itself?
- Can architectural changes alone achieve compute-optimal per-prompt scaling?
- How does algorithmic control flow define computational graph structure in LLM programs?
- Can algorithmic control flow over prompts simulate traditional programming languages?
- Can optimization algorithms exploit the shift between procedural and planning bottlenecks?
- Does parallel task structure determine optimal multi-agent architecture?
- How should proportionality constraints be implemented in agentic systems?
- What makes planning, tool use, and reasoning into jointly optimizable subsystems?
- What makes composable abstractions emerge under performance pressure in agent systems?
- Can we design efficient agents by targeting constraints directly?
- Can structured protocols outperform pure emergence in autonomous multi-agent coordination?
- Why does structured protocol coordination outperform free-form agent-to-agent communication?
- Can construction-time routing and runtime agent pruning be combined effectively?
- Can multimodal agents use entity-centric graphs within this three-axis framework?
- How do planning and memory compress agentic system costs?
- How do multi-agent routers balance flexibility against interpretability in design?
- What structural constraints produce recursion costs in agentic systems?
- Which task requirements does each graph view address in agent systems?
- How do context engineering limits relate to multi-agent coordination problems?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can reasoning topologies be formally classified as graph types?
This explores whether Chain of Thought, Tree of Thought, and Graph of Thought represent distinct formal graph structures with different computational properties. Understanding this matters because the topology itself determines what reasoning strategies are possible.
the individual-reasoning analog; this note extends the graph formalism to agent systems
-
Do reasoning cycles in hidden states reveal aha moments?
What if the internal loops in model reasoning—visible in hidden-state topology—correspond to the reconsidering moments that happen during reasoning? This note explores whether graph cyclicity captures a mechanistic signature of insight.
topology optimization for reasoning; this note demonstrates topology optimization for agent orchestration
-
Can extreme task decomposition enable reliable execution at million-step scale?
Can breaking tasks into maximally atomic subtasks with voting-based error correction solve the fundamental reliability problem in long-horizon tasks? This challenges whether better models or better decomposition is the path to high-reliability AI systems.
MAKER as a specific composite-graph pattern
-
Can multiple LLMs coordinate without explicit collaboration rules?
When multiple language models share a concurrent key-value cache, do they spontaneously develop coordination strategies? This matters because it could reveal how reasoning models naturally collaborate and inform more efficient parallel inference.
emergent coordination as a graph property
-
Can algorithms control LLM reasoning better than LLMs alone?
Explores whether embedding LLMs within algorithmic control flow—where programs manage state and context filtering—enables complex task decomposition beyond what LLMs achieve through self-managed reasoning chains.
LLM Programs are computational graphs with predetermined topology: the algorithm defines the edges, each LLM call is a node, and information hiding enforces the graph structure; the difference is that LLM Programs fix the graph at design time while optimizable graphs discover it through optimization
-
Can modular cognitive tools unlock reasoning without training?
Can reasoning capabilities be elicited by structuring LLM calls as isolated cognitive operations—understanding, recalling, examining, and backtracking—rather than through reinforcement learning?
cognitive tools are specific node implementations within the computational graph: each tool (understand, recall, examine, backtrack) is a sandboxed function node that can be composed into agent-level reasoning graphs
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Language Agents as Optimizable Graphs
- AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems
- Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence
- FlowReasoner: Reinforcing Query-Level Meta-Agents
- Graph-enhanced Large Language Models in Asynchronous Plan Reasoning
- Scaling Behavior of Single LLM-Driven Multi-Agent Systems
- AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
- FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Original note title
language agents as optimizable computational graphs unify prompting techniques and enable automatic optimization of both prompts and agent orchestration