Which Algorithm Specification Formats Help Language Models Implement Machine Learning Algorithms?

Masahiro Kato and Taka Kato. 2026. Preprint (arXiv:2607.03158). 日本語版

When an LLM implements an algorithm from a research paper, how much does its accuracy depend on the way the specification is written? We wrote the same machine learning algorithms in seven formats and measured, over 4,020 generated implementations, whether an LLM implements them correctly in a single attempt. This study supplies experimental grounds for the AI-Ready practice of preparing documents and algorithms in a form that AI can reuse.

The problem

Having an LLM implement a method from a paper is now common practice. Papers often leave the implementation choices implicit, and the generated code goes wrong in the details. How the results change when the specification is written as prose, as pseudocode, or in a structured format had not been measured systematically.

The proposed method

We prepared the specification of the same algorithm in seven formats: ordinary prose, pseudocode in the style of the LaTeX algorithm environment, pseudocode as if extracted from a PDF, a Markdown outline, YAML-like, JSON-like, and Python skeleton code. Crossing five machine learning tasks with three models and four experimental settings that differ in how much information is supplied yields 4,020 generated implementations. We graded them with hidden tests that probe the details deciding correctness, such as the order of equivalent inputs, array shapes, numerical conventions, the structure of return values, and behavior on invalid input.

Main results and conditions

When only part of the information needed for implementation was supplied, the format mattered a great deal. Pseudocode in the style of the LaTeX algorithm environment had the largest average effect, followed by YAML-like specifications and by prose. When the necessary information was all present, the differences by format disappeared for GPT-5.4 mini, while they persisted for smaller models such as Gemma 3 4B and Llama 3.2 3B. Skeleton code that supplies the function signature did not yield a consistent improvement. Taken together, the results say that whether the information determining the implementation is written down matters more than how the specification looks. The targets were five function-scale machine learning algorithms and a particular set of models, so this experiment does not support claims about implementing large systems.

Relation to earlier work

The evaluation of code generation by LLMs was put on a firm footing by Chen and coauthors, who measured function-level implementation (2021, Evaluating Large Language Models Trained on Code), and by Austin and coauthors, who studied program synthesis (2021, Program Synthesis with Large Language Models). That the surface form of a prompt changes performance was shown by Sclar and coauthors (2023, Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design). We add to this line by fixing the object of evaluation as specifications of machine learning algorithms, and by measuring differences of surface form separately from differences in how much information is given.

Where it applies

If you want AI to implement the methods of a research group, or want internal specifications of algorithms and models written so that AI does not misread them, the results say which information to state explicitly. The wider picture of preparing such documents is described on the AI-Ready Knowledge Infrastructure page.

Paper and materials

BibTeX

@misc{algorithm-specification-formats-llm,
  author       = {Masahiro Kato and Taka Kato},
  title        = {Which Algorithm Specification Formats Help Language Models Implement Machine Learning Algorithms?},
  year         = {2026},
  eprint       = {2607.03158},
  archivePrefix = {arXiv},
  url          = {https://arxiv.org/abs/2607.03158},
}