SYNTHESIS NOTE
Do larger language models solve constrained optimization better?
Explores whether scaling LLMs—through more parameters, better training, or reasoning extensions—improves their ability to satisfy constraints in real optimization problems like power grids and portfolios.
When evaluated on real constrained-optimization problems — optimal power flow, financial portfolio constraints, cyber-security feasibility — LLMs cluster around 55-60% constraint satisfaction across virtually all conditions tested. The plateau is robust to changes in architecture, parameter count, and training regime. Reasoning models, despite extended chain-of-thought, do not systematically beat their non-reasoning counterparts on these tasks.
The flatness of the plateau is the finding. Most LLM capability work assumes that the relevant axis is performance vs scale, and that closing a gap is a matter of training on more or better data. Constrained optimization does not behave that way. The benchmark distinguishes problems that require jointly interpreting structured input, doing multi-step arithmetic, satisfying interacting physical constraints, and converging to feasible solutions. On the joint task, the model class itself appears to be near a ceiling.
This is distinct from general reasoning benchmarks (MMLU, GPQA) and from logical reasoning benchmarks (ARC-AGI, SATBench, ZebraLogic). Those measure either broad knowledge or synthetic constraint puzzles. Real engineering optimization requires the model to execute iterative numerical procedures over physical constraints, and that procedural execution is where the plateau lives.
The deployment implication is sharp: telling executives that "LLMs will optimize the grid" or "LLMs will solve constrained portfolio problems" is currently an overclaim. The same finding suggests the productive direction is not "wait for the next model" but "change the paradigm" — restrict the LLM to abstraction tasks and hand numeric work to solvers.
Inquiring lines that read this note 122
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does alignment training create genuine alignment or just output compliance? Why do stronger reasoning capabilities create tradeoffs with instruction following?- Why do only two of fourteen models improve when problem constraints are removed?
- How do unstated feasibility constraints affect model decision-making?
- Can explicit constraint statements override the dominance of surface heuristics?
- Why do reasoning models fail to improve constrained optimization performance?
- Can models be trained to verify feasibility before proposing plans?
- When does the right constraint beat additional model capacity?
- How does inference compute substitution affect the training parameter scaling trade-off?
- Does trading model size for inference steps improve overall efficiency scaling?
- Can non-variational posterior approximation schemes deliver comparable reasoning improvements?
- Can closed-form solutions compete with gradient descent optimization?
- Can language models execute iterative numerical methods in latent space?
- What structural constraints matter more than model depth for CF?
- Do tool-enabled reasoning models close the gap on constraint satisfaction?
- Would hybrid systems combining LLMs with symbolic solvers overcome the retraction limitation?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Why does genetic programming outperform direct LLM generation by 86 percent?
- Can optimization algorithms exploit the shift between procedural and planning bottlenecks?
- What planning tasks benefit most from combining LLM generation with external verification?
- Can the LLM-Modulo framework extend solver integration to domain planning?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- Can LLMs successfully translate natural language into formal solver specifications?
- How should organizations redesign workflows if LLMs cannot solve optimization directly?
- What concrete problems do LLMs solve at the computational level?
- What makes natural-language APIs particularly suited to LLM-based simulation?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- What constraint satisfaction rate do LLMs achieve at scale?
- Can LLMs simultaneously reason and optimize their own modules?
- Does constraint-setting before generation change what LLMs can contribute?
- Can surface heuristics override implicit constraints in domain-specific reasoning?
- Why do macro and micro forecasting scales require different reasoning approaches?
- Can a tiny recursive network beat billion-parameter models on hard problems?
- How do unstated constraints become invisible to training data distributions?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- What power-law scaling patterns emerge when consistency models are trained at scale?
- What production constraints should determine paradigm selection?
- Can routing enable heterogeneous SLM-first architectures at scale?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Do small models show different parameter efficiency patterns than large models?
- What makes routing a better investment than training larger models?
- How do constrained versus unconstrained domains flip LLM novelty patterns?
- What explains the 87 percent to 12 percent cliff in plan executability?
- How do LLMs compress specific expert knowledge into median abstraction?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Do LLMs fail exploration because of context integration or computational limitations?
- Why do LLMs fail at directly solving stochastic control problems?
- Can we systematically enumerate LLM failure modes from first principles?
- Why do LLMs fail at iterative numerical computation in latent space?
- What capability boundary exists in LLM prediction of effect sizes?
- Can universal function approximators be expensive to learn in practice?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Why does scaling data and model size improve compositional generalization?
- Why do intermediate LLM layers become more precise in frontier models?
- Does scaling model size solve compositional generalization problems?
- Can smaller models actually perform well on specific downstream tasks?
- How do general language model benchmarks predict specialized domain performance?
- Why do standard NLP benchmarks hide the most critical language limitations?
- Why do different LLMs converge on nearly identical outputs?
- How should tiny language models be architected differently than large ones?
- Why do language models plateau at 55 to 60 percent constraint satisfaction?
- Why do language models fail at iterative numerical optimization despite scale?
- Why do language models plateau at constraint satisfaction regardless of scale?
- How do parameter scaling and latent vectors interact in language models?
- How does nesting optimization levels improve on traditional network depth?
- How do search and reasoning workflows improve forecasting performance over base models?
- Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness?
- How should inference budget adapt based on problem difficulty?
- Can compute allocation and model routing be combined for better results?
- Can weaker models match stronger ones with sufficient search and reasoning budget?
- Why do energy-based models generalize better on out-of-distribution data than standard transformers?
- What formal language complexity level matches transformer computational limits best?
- What limits the effectiveness of formal language pretraining on transformer architectures?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Does the optimal model size depend on what capabilities you actually need?
- Why do production systems optimize for three model classes instead of foundation models?
- Can models adapt and combine search strategies beyond their training algorithm?
- Does fine-tuning a small model match fine-tuning a large one?
- How does fitness-proportional selection guide LLM recombination in unstructured solution spaces?
- Can LLM-based crossover and mutation work in unstructured natural language spaces?
- What decomposition level minimizes both error rate and computational cost in practice?
- What planning strategies reduce execution steps without sacrificing solution quality?
- Why do task-specific heuristics fail at generalizing to sparse data regions?
- Can smaller specialist models outperform large generalist models on domain tasks?
- How do LLM activations sparsify differently under out-of-distribution inputs?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- What makes certain bond distributions more learnable than others?
- Can scaling data alone solve performance gaps on long-tail concepts?
- How do codebook length and data similarity affect accuracy in LLM coding?
- Can LLMs improve at simple deduction through different training approaches?
- Why do smaller LLMs fail at zero-shot argument scheme classification?
- How does constraint complexity relate to optimal reasoning token budgets?
- What limits external scaling when a model lacks reasoning foundation?
- What makes fixed-point convergence better than learned halt tokens?
- Can a model be strong at MMLU but weak at long-horizon tasks?
- Can tool use or self-conditioning fix degradation in extended LLM workflows?
- Can LLM-synthesized behavioral heuristics compete with learned policy improvements?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do larger language models solve constrained opti…
Do reasoning models actually beat standard models …
Do large language models actually perform iterativ…
Should LLMs handle abstraction only in optimizatio…
Does more thinking time always improve reasoning a…
Can large language models translate natural langua…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do reasoning models actually beat standard models on optimization?
Explores whether extended chain-of-thought in reasoning models delivers performance gains on constraint-satisfaction problems like power-grid optimization. Matters because reasoning models are treated as automatic upgrades, but the evidence may not support that claim.
same paper, the reasoning-model specific finding
-
Do large language models actually perform iterative optimization?
Explores whether LLMs execute genuine numerical procedures like Newton-Raphson or instead pattern-match to memorized solution templates when solving constrained optimization problems.
same paper, the mechanism for the plateau
-
Should LLMs handle abstraction only in optimization?
What if LLMs worked exclusively on translating problems to formal constraints, while deterministic solvers handled the numeric work? Explores whether this division of labor could overcome LLM failures in iterative computation.
same paper, the proposed solution
-
Does more thinking time always improve reasoning accuracy?
Explores whether extending a model's thinking tokens linearly improves performance, or if there's a point beyond which additional reasoning becomes counterproductive.
adjacent: chain-of-thought has its own ceiling
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
adjacent: NL → formal translation limits
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can Large Language Models Reason and Optimize Under Constraints?
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Distilling LLMs' Decomposition Abilities into Compact Language Models
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models
- 𝙻𝙼𝟸: A Simple Society of Language Models Solves Complex Reasoning
Original note title
LLMs plateau at 55 to 60 percent constraint satisfaction on genuine optimization regardless of scale architecture or training