Editing 1,023 weights raised Qwen3.5-0.8B's MATH-500 score by 29 points. Most of the paper is about why that rarely works

A paper posted to arXiv this month by researchers listed under Shanghai Artificial Intelligence Laboratory, Tongji University and Fudan University reports that a reasoning model raised Qwen3.5-0.8B’s held-out MATH-500 score from 8.35% to 37.80% by changing 1,023 values in its weights. That is roughly one parameter in 800,000, changed with no gradients and no optimizer.

The number will travel well, but the paper’s more useful contribution is the denominator. It is a broad stress test of Weight-Space Programming (WSP), the idea introduced on November 2 by Takehi, Teoh, Zhang and Khattab at MIT CSAIL, and across its 16 edit trajectories the method looks narrower and noisier than its first demonstration suggested.

What the agent actually does

WSP gives a general-purpose LLM access to another network’s raw tensors and asks it to reason about them and write new values directly. Here the editor is GPT-6 Astra at maximum reasoning effort, and the targets are Qwen3.5-0.8B and Qwen3.5-9B. Each search gets a budget of 50 successful forward requests, which can be generation or teacher-forced passes, plus up to 300 Python analyses. Edits are restricted to transformer-block parameters. The agent cannot write a training loop.

Budget matters more than it might appear. The MIT post showed that with 10 or 25 forward passes its edited model barely improved: MATH-500 reached 12.3% and 14.1%. At 50 passes it jumped to 48.1%. WSP therefore behaves less like a one-shot act of reading weights and more like interpretability-driven debugging with a strict evaluation allowance.

The new paper evaluates against seven benchmarks: MATH-500, MMLU-Pro, IFEval, GPQA Diamond, MedXpertQA, SciBench and SciCode. Every comparison is against the unedited checkpoint under matched sampling settings. It does not compare against gradient-based fine-tuning, so nothing here says WSP beats a LoRA on cost or quality.

The results, including the failures

On Qwen3.5-0.8B, held-out gains were large where the base model was weak:

• MATH-500: 8.35% to 37.80% (+29.45 points)
• IFEval: 21.40% to 58.12% (+36.72)
• SciBench: 1.38% to 18.49% (+17.11)
• GPQA Diamond: 25.25% to 29.67% (+4.42)
• SciCode: 0.00% to 0.00%

On Qwen3.5-9B, a model already scoring near the ceiling on some tasks, the picture changes. IFEval went from 89.30% to 91.14%, SciBench from 72.46% to 75.49%, and SciCode from 1.54% to 3.08%. A math candidate that showed +29.69 points on development data landed at 97.35% against a 97.70% baseline on the held-out test, so the agent’s apparent win vanished. In six of the 16 trajectories, the editor kept the original weights because its candidates failed validation.

The same pattern shows up on the small model. A MedXpertQA edit that gained 17.19 points during discovery produced +0.24 on held-out data, moving from 10.57% to 10.81%. The authors state plainly that development results were selected adaptively and carry no correction for multiple comparisons. Anyone reading a WSP discovery number should treat it as a hypothesis until it survives held-out testing.

Reproducibility of the path is also weak even when the outcome holds. Three independent math searches on the same model produced +29.45 points with 1,023 changed values, +5.00 with 1,037,572 values, and +27.30 with 1,981,083 values. The edits differ by three orders of magnitude in size. Two of them reach similar performance, which suggests there are many ways to reach this behavior change, and the third shows the search can also settle for much less.

Why such a small edit can matter so much

The strongest evidence that the 1,023-value edit is targeted comes from its controls. The edit lies in a single output column of the layer-23 MLP down-projection. At an 8,192-token cap, on a batch of 48 questions, the base model got 6 right, the targeted edit 20, ablating the column 4, reversing the direction 0, and a random orthogonal direction of equal norm 10. A dose sweep at the 32,768-token cap gave 7, 22, 25, 28, 21 and 18 correct across increasing strengths, peaking at the chosen setting. Confirmation on six disjoint development sets, 1,152 questions in total, moved correct answers from 126 to 484, with 377 rescues and only 19 regressions. Those batches are small and the random control is a single direction, but a dose-response curve that rises and falls is what a real causal handle tends to look like.

What that handle controls is the critical question. In the MIT work, 93% of the base model’s MATH-500 answers ran to the 32,768-token limit, and the edit cut that to 33.1%. Qwen3.5-0.8B was stuck in reasoning loops that its own model card warns about. The MIT authors then tried the same task on Gemma 3 270M, where only 8.5% of answers hit the limit, and Astra could not move MATH-500 by more than about a point. Their conclusion was that editing could remove an obstacle hiding an ability, but “could not add an ability the model lacked.”

The Shanghai results fit that reading. On a separate AIME challenge subset, both the base and the edited models scored 0 of 64. That is consistent with an edited model recovering problems it could already solve once it stopped looping, without gaining the ability to solve hard competition math. Critics of the original result have made the same point: the jump is better described as a loop fix than as math learning.

There is a subtle complication, which the paper flags: accuracy and truncation can move in opposite directions. One MedXpertQA candidate cut truncations from 502 to 296 while accuracy slipped from 1,376 to 1,356 out of 3,072. Reducing looping does not automatically produce better answers.

What this means for people who ship models

The practical value today is as a debugging tool for small, cheap open-weight models with a known and diagnosable defect. Pathological repetition is a good candidate: a fix that changes a few thousand values is cheap to store and audit, and the edit can be expressed as a small diff against public weights. Compared with fine-tuning methods such as the supervised fine-tune behind Nous Research’s Arete-27B, the appeal is precision. The cost is that every candidate must be validated with repeated generations, which the authors describe as expensive.

Transfer is the most interesting positive signal. Without further editing, the math-edited 0.8B model also improved on GPQA Diamond by 3.03 points, IFEval by 12.18 and SciBench by 20.10. That is consistent with fixing a general failure mode. It also means an edit made for one purpose changes behavior elsewhere, which is a governance concern. An edit this small is hard to spot in a large checkpoint, so hashes and provenance matter more once weights can be meaningfully changed in a few thousand values.

What to watch

The authors list open problems: how to select interventions reliably, how to evaluate more cheaply, how multiple edit locations interact, and why these edits work at all. WSP also has not been systematically tested on longer-horizon reasoning or agentic tasks, which is where the economic interest lies.

The test that would change the assessment is a demonstration on a model that does not have an obvious pathology, measured against a fine-tuning baseline on equal compute. Until that exists, the fair summary of this paper is that an LLM can sometimes find and repair a specific fault in another network’s weights, and that in more than a third of these attempts, by its own validation, it found nothing worth keeping.