Novel LLM Architectures
Related topics:
- A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic SystemsTo address this limitation, recent research has explored agent evolution techniques that aim to automatically enhance agent systems based on interaction data and environmental feedback. This emerging …
- A Survey on Diffusion Language ModelsAbstract—Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm. By generating tokens in parallel through an iterati…
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning1 Introduction Reinforcement learning (RL) has emerged as a new scaling paradigm for enhancing the capabilities of large language models (LLMs) by enabling thinking abilities [52]. Given a prompt, RL…
- Agent-Centric Projection of Prompting Techniques and Implications for Synthetic Training Data for Large Language ModelsThis position paper introduces and explains the concepts of linear contexts (a single, continuous sequence of interactions) and non-linear contexts (branching or multi-path) in LLM systems. These conc…
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMsIn this paper, we introduce a novel learning paradigm for adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either …
- AlphaGo Moment for Model Architecture DiscoveryWhile AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity, creating an increasingly severe development bott…
- Base Models Know How to Reason, Thinking Models Learn WhenWhy do thinking language models like DeepSeek R1 outperform their base counterparts? Despite consistent performance gains, it remains unclear to what extent thinking models learn entirely new reasonin…
- Beyond Turing: Memory-Amortized Inference as a Foundation for Cognitive ComputationAbstract—Intelligence is fundamentally non-ergodic: it emerges not from uniform sampling or optimization from scratch, but from the structured reuse of prior inference trajectories. We introduce Memor…
- Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We NeedLanguage models traditionally utilized for cross-domain generalization in natural language understanding and generation have recently demonstrated task-specific reasoning through inference-time scalin…
- Byte Latent Transformer: Patches Scale Better Than TokensWe introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inferen…
- Cognitive Architectures for Language Agents“We introduce such a framework, drawing parallels with two ideas from the history of computing and artificial intelligence (AI): production systems and cognitive architectures. Production systems gene…
- Conversational Graph Grounded Policy Learning for Open-Domain Conversation GenerationTo address the challenge of policy learning in open-domain multi-turn conversation, we propose to represent prior information about dialog transitions as a graph and learn a graph grounded dialog poli…
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving AgentsMost of today’s AI systems are constrained by human-designed, fixed architectures and cannot autonomously and continuously improve themselves. The scientific method, on the other hand, provides a cumu…
- Deep Researcher with Test-Time DiffusionDeep research agents, powered by Large Language Models (LLMs), are rapidly advancing; yet, their performance often plateaus when generating complex, long-form research reports using generic test-time …
- DeepNet: Scaling Transformers to 1,000 LayersIn this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DEEPNORM) to modify the residual connection i…
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsWe introduce DeepSeek-V3.2, a model that harmonizes high computational efficiency with superior reasoning and agent performance. The key technical breakthroughs of DeepSeek-V3.2 are as follows: (1) De…
- Dialogue State Tracking with a Language Model using Schema-Driven PromptingTask-oriented conversational systems often use dialogue state tracking to represent the user’s intentions, which involves filling in values of pre-defined slots. Many approaches have been proposed, of…
- Diffusion-LM Improves Controllable Text GenerationControlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sente…
- Efficient Streaming Language Models with Attention SinksDeploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during …
- Emergent Hierarchical Reasoning In LLMs Through Reinforcement LearningReinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaq…
- End-to-End Test-Time Training for Long ContextOn the other hand, Transformers with self-attention still struggle to efficiently process long context equivalent to years of human experience, in part because they are designed for nearly lossless re…
- Energy-Based Transformers are Scalable Learners and ThinkersInference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limita…
- Evolving Deeper LLM ThinkingWe explore an evolutionary search strategy for scaling inference time compute in Large Language Models. The proposed approach, Mind Evolution, uses a language model to generate, recombine and refine c…
- Extrapolation by Association: Length Generalization Transfer in TransformersTransformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this pa…
- Fast, Slow, and Tool-augmented Thinking for LLMs: A ReviewLarge Language Models (LLMs) have demonstrated remarkable progress in reasoning across diverse domains. However, effective reasoning in real-world tasks requires adapting the reasoning strategy to the…
- From Language to Logic: A Bi-Level Framework for Structured ReasoningStructured reasoning over natural language inputs remains a core challenge in artificial intelligence, as it requires bridging the gap between unstructured linguistic expressions and formal logical re…
- Generalization to New Sequential Decision Making Tasks with In-Context LearningHowever, the sequential decision making setting poses additional challenges having a lower tolerance for errors since the environment’s stochasticity or the agent’s actions can lead to unseen, and som…
- Guidance is All You Need: Temperature-Guided Reasoning in Large Language ModelsWe present Quasar-1, a novel architecture that introduces temperature-guided reasoning to large language models through the Token Temperature Mechanism (TTM) and Guided Sequence of Thought (GSoT). Our…
- HiTKG: Towards Goal-Oriented Conversations via Multi-Hierarchy LearningThe existing recurrent and graph attention based KG walkers either insufficiently utilize the conversation states or lack global guidance. In our work, a hierarchical model learns goal planning in a h…
- HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web SearchesRecently, large reasoning models have demonstrated strong mathematical and coding abilities, and deep search leverages their reasoning capabilities in challenging information retrieval tasks. Existing…
- Hierarchical Reasoning ModelReasoning, the process of devising and executing complex goal-oriented action sequences, remains a critical challenge in AI. Current large language models (LLMs) primarily employ Chain-of-Thought (CoT…
- HyperagentsSelf-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes. Existing approaches to recursive self-improvement typical…
- Insert-expansions For Tool-enabled Conversational Agents“This paper delves into an advanced implementation of Chain-of-Thought-Prompting in Large Language Models, focusing on the use of tools (or "plug-ins") within the explicit reasoning paths generated by…
- It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online OptimizationDesigning efficient and effective architectural backbones has been in the core of research efforts to enhance the capability of foundation models. Inspired by the human cognitive phenomenon of attenti…
- Jamba: A Hybrid Transformer-Mamba Language ModelWe present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba interleaves blocks of Transformer and Mamba layer…
- LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationwe propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust…
- Language Modeling by Language ModelsCan we leverage LLMs to model the process of discovering novel language model (LM) architectures? Inspired by real research, we propose a multi-agent LLM approach that simulates the conventional stage…
- Large Causal Models From Large Language ModelsWe introduce a new paradigm for building large causal models (LCMs) that exploits the enormous potential latent in today’s large language models (LLMs). We describe our ongoing experiments with an imp…
- Large Language Diffusion ModelsIs the autoregressive paradigm the only viable path to achieving the intelligence exhibited by LLMs? we argue that scalability is primarily a consequence of the interplay between Transformers (Vaswan…
- Large Language Model ProgramsIn recent years, large pre-trained language models (LLMs) have demonstrated the ability to follow instructions and perform novel tasks from a few examples. The possibility to parameterise an LLM throu…
- Latent Collaboration in Multi-Agent SystemsMulti-agent systems (MAS) extend large language models (LLMs) from independent single-model reasoning to coordinative system-level intelligence. While existing LLM agents depend on text-based mediatio…
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language ModelsLinear attention is an efficient attention mechanism that has recently emerged as a promising alternative to conventional softmax attention. With its ability to process tokens in linear computational …
- Looking beyond the next tokenThe structure of causal language model training assumes that each token can be accurately predicted from the previous context. This contrasts with humans’ natural writing and reasoning process, where …
- Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth TransformersLarge language models (LLMs) (Brown et al., 2020) are known to acquire substantial factual knowledge during pretraining, storing it in their parameters (Geva et al., 2023). However, how effectively th…
- MatFormer: Nested Transformer for Elastic InferenceFoundation models are applied in a broad spectrum of settings with different inference constraints, from massive multi-accelerator clusters to resource-constrained standalone mobile devices. However, …
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsLarge Language Models (LLMs) employ autoregressive decoding that requires sequential computation, with each step reliant on the previous one’s output. This creates a bottleneck as each step necessitat…
- Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents“Large Language Models (LLMs) are currently capable of generating human-like responses in open-domain tasks [4]. This has led to a new generation of conversational agents, such as chatGPT, which are n…
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning AttentionWe introduce MiniMax-M1, the world’s first open-weight, large-scale hybrid-attention reasoning model. MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning …
- Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They SayOpen-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LL…
- Multi-Token AttentionSoft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and …
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionLong-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offe…
- Nested Learning: The Illusion of Deep Learning Architecture ExpandedIn the previous sections, we discussed the concept of nested learning and how existing well-known components of neural networks such as popular optimizers and architectures fall under the NL paradigm.…
- Nested Learning: The Illusion of Deep Learning ArchitecturesOver the last decades, developing more powerful neural architectures and simultaneously designing optimization algorithms to effectively train them have been the core of research efforts to enhance th…
- Neurosymbolic AI- Why, What, and HowHuman perception-inspired machine perception, in the context of AI, refers to large-scale pattern recognition from raw data using neural networks trained using self-supervised learning objectives such…
- Post-Completion Learning for Language ModelsCurrent language model training paradigms typically terminate learning upon reaching the end-of-sequence (<eos) token, overlooking the potential learning opportunities in the post-completion space. We…
- Pushdown Layers: Encoding Recursive Structure in Transformer Language ModelsRecursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer langua…
- Reasoning Language Models: A Blueprintsuch as OpenAI’s o1 and o3, DeepSeek-V3, and Alibaba’s QwQ, have redefined AI’s problem-solving capabilities by extending large language models (LLMs) with advanced reasoning mechanisms. Yet, their hi…
- Recursive Language ModelsWe study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive Language Models (RLMs), a general inference strategy…
- Reinforcement Pre-TrainingIn this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a rea…
- Representation biases: will we achieve complete understanding by analyzing representations?A common approach in neuroscience is to study neural representations as a means to understand a system—increasingly, by relating the neural representations to the internal representations learned by c…
- Rethinking Thinking Tokens: LLMs as Improvement OperatorsReasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accur…
- Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-upwe propose Reversal of Thought (RoT), a novel framework aimed at enhancing the logical reasoning abilities of LLMs. RoT utilizes a Preference-Guided Reverse Reasoning warm-up strategy, which integrate…
- SPICE: Self-Play In Corpus Environments Improves ReasoningSelf-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts …
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth ApproachWe study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling…
- Self-Discover: Large Language Models Self-Compose Reasoning Structures*Table 2. All 39 reasoning modules consisting of high-level cognitive heuristics for problem-solving. We adopt them from Fernando et al.* (_2023_). Reasoning Modules 1 How could I devise an experim…
- Solving a Million-Step LLM Task with Zero ErrorsLLMs have achieved remarkable breakthroughs in reasoning, insights, and tool use, but chaining these abilities into extended processes at the scale of those routinely executed by humans, organizations…
- The Consensus Game: Language Model Generation via Equilibrium SearchWhen applied to question answering and other text generation tasks, language models (LMs) may be queried generatively (by sampling answers from their output distribution) or discriminatively (by using…
- The Future of AI: Exploring the Potential of Large Concept Modelsthese models are inherently limited by their token-level processing, which restricts their ability to perform abstract reasoning, conceptual understanding, and efficient generation of long-form conten…
- The Missing Layer of AGI: From Pattern Alchemy to Coordination PhysicsInfluential critiques argue that Large Language Models (LLMs) are a dead end for AGI: “mere pattern matchers” structurally incapable of reasoning or planning. We argue this conclusion misidentifies th…
- The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought ReasoningLarge language models (LLMs) often fail to learn effective long chain-of-thought (Long CoT) reasoning from human or non-Long-CoT LLMs imitation. To understand this, we propose that effective and learn…
- The Serial Scaling HypothesisWhile machine learning has advanced through massive parallelization, we identify a critical blind spot: some problems are fundamentally sequential. These "inherently serial" problems—from mathematical…
- The Vanishing Gradient Problem for Stiff Neural Differential EquationsNeural differential equations have become a transformative tool in machine learning and scientific computing, enabling data-driven modeling of complex, time-dependent phenomena in fields ranging from …
- Thinking Augmented Pre-trainingThis paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for …
- Thinking vs. Doing: Agents that Reason by Scaling Test-Time InteractionAbstract: The current paradigm of test-time scaling relies on generating long reasoning traces (“thinking” more) before producing a response. In agent problems that require interaction, this can be do…
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model ParametersTransformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a si…
- Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization AnalysisChain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability of large language models by augmenting the query using multiple examples with multiple intermediate steps. De…
- Transformer2: Self-adaptive LLMsSelf-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse…
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityWhile Transformers have been the main architecture behind deep learning’s success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transfor…