From Articles to Code: On-Demand Generation of Core Algorithms from Scientific Publications

ABSTRACT

Maintaining software packages imposes significant costs due to dependency management, bug fixes, and versioning. We show that rich method descriptions in scientific publications can serve as standalone specifications for modern large language models (LLMs), enabling on-demand code generation that could supplant human-maintained libraries. We benchmark state-of-the-art models (GPT-o4-mini-high, Gemini Pro 2.5, Claude Sonnet 4) by tasking them with implementing a diverse set of core algorithms drawn from original publications. Our results demonstrate that current LLMs can reliably reproduce package functionality with performance indistinguishable from conventional libraries. These findings foreshadow a paradigm shift toward flexible, on-demand code generation and away from static, human-maintained packages, which will result in reduced maintenance overhead by leveraging published articles as sufficient context for the automated implementation of analytical workflows.

INTRODUCTION

Scientific articles serve as the foundation of reproducible computational research, providing detailed, peer-reviewed descriptions of novel algorithms and methodologies. However, the journey from paper to robust, usable software is fraught with challenges. Translating narrative algorithmic insights into production-grade implementations remains a labor-intensive process, frequently hindered by ambiguities or omitted practical details in published work. This gap is widely recognized as a core barrier to scientific transparency and the credibility of computational findings, with reproducibility crises and failed software reimplementations demonstrating the difficulties researchers face in reusing published work.1,2

To bridge this divide, software libraries provide powerful abstractions, encapsulating complex scientific and statistical methods behind user-friendly APIs. It is well accepted that code libraries can enable reproducible research2,3. While these libraries accelerate research, their maintenance is a monumental task. Key challenges include:

● Prominent and deeply interwoven dependency chains, where updates in foundational packages propagate breaking changes throughout the ecosystem, creating a delicate balance between innovation and stability4;

● Subtle edge-case bugs that require targeted, labor-intensive fixes, often only identified through extensive community usage and evolving scientific requirements;

● Version mismatches and a lack of standardized compute environments undermine the reproducibility of scientific research.

Recent advances in large language models (LLMs) for code generation, such as OpenAI Codex5 and DeepMind AlphaCode6, mark a paradigm shift in how computational methods can be instantiated. These models, trained on billions of lines of code and natural language, can translate natural-language problem descriptions into executable software. In several benchmarks, these models not only accelerate development but also demonstrate competitive performance in real-world tasks, with capabilities advancing rapidly across programming languages and domains. However, success rates vary: while straightforward problems are effectively solved, multi-step scientific workflows and nuanced research tasks remain a challenge for current LLMs5,6.

When paired with retrieval-augmented generation (RAG) frameworks7, LLMs can fetch precise algorithmic details or API documentation from curated corpora, including scientific articles and official documentation, at inference time. This approach enables a model to synthesize code directly guided by original research, rather than relying solely on historical training data. By dynamically integrating this external knowledge just-in-time, the distinction between published algorithm and runnable implementation begins to fade, hinting at a new paradigm where articles themselves operate as executable specifications.

Random Forest. The original PDF of the paper8 was attached, and the prompt was: “Forget all our previous chats and forget everything you know about the random forest method for machine learning. Using only the information in the attached PDF, try your hardest to create an implementation of a random forest that exactly matches the algorithm for classification. Your function should work with the iris dataset from sklearn to classify the flower type. You only get one try to do this correctly, so please try very carefully to get the code correct.” The code is in RF-compare-models.ipynb.