Ying Wang

LM: A Programming Language Where LLMs Write Fewer Bugs

·Ying Wang
lmprogramming-languagellmbenchmarkrust

Every programming language is a tradeoff between expressiveness and safety. JavaScript lets you write "1" + 2 and get "12". Python silently returns None from functions that forget to return. C lets you read past array boundaries.

Humans learn to navigate these traps through experience. LLMs don't have experience — they have patterns. And patterns break at the edges.

LM is a programming language designed so that common LLM coding mistakes become compile errors. No mutation, no null, no exceptions, no implicit type conversions, no inheritance. The compiler speaks both human-readable and JSON diagnostic formats, so LLM agents can parse errors and self-correct.

The hypothesis: on a standard set of programming tasks, an LLM writing LM should achieve higher first-pass correctness than the same LLM writing TypeScript or Python.

The Experiment

We tested this with a blind benchmark:

  • 30 standard tasks + 10 adversarial tasks designed to trigger common LLM mistakes
  • Three languages: LM, TypeScript, Python
  • Two models: Claude Opus 4.6 (strongest) and Claude Haiku 4.5 (weakest)
  • Blind generation: each model saw only the task description, never the expected output
  • Solutions were generated by independent agents with no shared context

The adversarial tasks target areas where LM's design should help:

Task Trap LM's Protection
String/number concat JS implicit conversion ++ for strings only, + for numbers only
Nested Option handling Forgetting null checks Must match to extract from Option
4-variant pattern match Missing a case Exhaustive matching enforced by compiler
3-step error chain Forgetting to propagate Must match every Result
Int/Float arithmetic Silent type coercion Int + Float is a compile error
Recursive ADT eval Wrong case in switch Exhaustive matching on all variants
Pure/IO separation Side effects in logic io effect system, compiler-enforced
Multi-field validation Error swallowing Result propagation required

Results

Strong Model (Claude Opus 4.6)

On the standard 30 tasks, all three languages performed equally well:

Language Standard (30) Adversarial (10)
LM 29/30 10/10
TypeScript 30/30 10/10
Python 29/30 10/10

No meaningful difference. Opus writes correct code in all three languages.

Weak Model (Claude Haiku 4.5)

This is where it gets interesting:

Language Adversarial (10)
LM 10/10
TypeScript 9/10
Python 9/10

Both TypeScript and Python failed on the same task: recursive expression evaluation with negation. Haiku wrote an Expr type with four variants (Num, Add, Mul, Neg) and got the Neg case wrong in both languages — the evaluator returned 8 instead of -3 for Neg(Num(3)).

In LM, the same model got it right. Why?

Why LM Helped

The failing TypeScript code looked something like this:

function evalExpr(e: Expr): number {
  switch (e.type) {
    case "Num": return e.value;
    case "Add": return evalExpr(e.left) + evalExpr(e.right);
    case "Mul": return evalExpr(e.left) * evalExpr(e.right);
    case "Neg": return evalExpr(e.expr);  // BUG: forgot to negate
  }
}

The LM version:

fn eval(e: Expr) -> Int {
    match e {
        Num(n) -> n,
        Add(a, b) -> eval(a) + eval(b),
        Mul(a, b) -> eval(a) * eval(b),
        Neg(x) -> 0 - eval(x),
    }
}

Two things happened:

  1. Exhaustive pattern matching — LM's compiler would reject the code if any variant was missing. The model had to explicitly handle Neg.

  2. Pattern destructuring — In LM, Neg(x) directly binds x to the inner expression. The model then had to write an expression using x that returns an Int. The structure of the pattern match guided the model toward the correct implementation: you've extracted x, now what do you do with it? Negate it.

In TypeScript, the switch/case pattern doesn't enforce this. The model wrote return evalExpr(e.expr) without the negation, and TypeScript happily accepted it.

The Documentation Lesson

The more surprising finding was how much LM's documentation matters.

Our first blind benchmark with LM scored only 18/30 — worse than TypeScript. Every failure was the same bug: the model defined helper functions inside other functions. LM doesn't support nested function definitions, but the language reference didn't mention this.

After adding one line to the docs — "No nested functions. All fn definitions must be at the top level." — the score jumped to 27/30. Two more documentation fixes brought it to 29/30.

Documentation version LM score
v1 (original) 18/30
v2 (+no nested functions) 27/30
v3 (+list_map is IO, +len vs str_len) 29/30

The language's constraints only work if the LLM knows about them. An undocumented constraint is worse than no constraint — the model assumes the feature exists and writes code that won't compile.

This is a general principle: for LLM-targeted languages, documentation completeness is a feature, not a nicety. Every constraint must be explicit. Every builtin's type signature must be listed. Every common mistake must have a "Wrong → Right" example.

What Didn't Work

The adversarial tasks were designed to trigger LLM mistakes in TypeScript and Python. In practice, only one task actually differentiated. The strong model made zero mistakes in any language. The weak model only failed on one task.

This tells us: LLM coding ability is improving faster than language design can exploit. The bugs LM was designed to prevent — null pointer errors, missing error handling, implicit type coercion — are bugs that today's best models rarely make.

Where LM might matter more:

  • Much weaker models — open-source 7B-13B models that make these mistakes frequently
  • Longer programs — 100+ line programs where accumulated small errors compound
  • Self-correction loops — LM's structured JSON diagnostics let agents parse errors and fix them automatically

The Language

LM is implemented in Rust: hand-written lexer, recursive descent parser, Hindley-Milner type inference, tree-walking interpreter. 182 tests, zero clippy warnings.

Rule Rationale
Complete immutability No aliasing bugs, no mutation surprises
+ for numbers, ++ for strings No "1" + 2 = "12" ambiguity
No null, use Option<T> Forces handling of absence
No exceptions, use Result<T, E> Error paths visible in type signatures
Exhaustive pattern matching Forget a case? Compile error
Effect system (pure / io) Pure functions can't do IO
No implicit conversion Int + Float = compile error

Try It

git clone https://github.com/yingwang/lm
cd lm
cargo build --release
./target/release/lmc run examples/hello.lm

The benchmark suite, all blind-generated solutions, and runner scripts are in benchmark/comparison/.

Conclusion

LM demonstrates that language design can improve LLM code generation quality — but the effect is modest with today's strongest models. The real gains come from compiler constraints that make wrong code uncompilable, complete documentation of every constraint, and structured diagnostics that let agents self-correct.

The strongest result: Claude Haiku writing LM achieved 10/10 on adversarial tasks where the same model writing TypeScript and Python both scored 9/10. The difference was a single bug — but it was exactly the kind of bug LM was designed to prevent.

Share

Comments

← Back to home