Research
Seven routes to a model’s weights.
Weight programming means producing or changing a network’s parameters by construction or by reasoning, instead of a long gradient-descent run on data. Seven routes exist. REWIRE puts them behind one verifiable workflow rather than inventing an eighth.
| Route | What it does | Examples |
|---|---|---|
| A · Compile | A program becomes exact transformer weights. Correct by construction, tiny in scale. | Tracr, ALTA, Percepta transformer-vm, AdderBoard |
| B · Generate | A task or document becomes a LoRA in one forward pass of a trained hypernetwork. | HyperNetworks, Text-to-LoRA, Doc-to-LoRA, Drag-and-Drop LLMs |
| C · Locate and edit | A fact is found in an MLP layer and rewritten in closed form, with side effects measured. | ROME, MEMIT, AlphaEdit, MEND, EasyEdit |
| D · Arithmetic and merge | Several models or deltas become one by weight-space algebra. | Task arithmetic, TIES, DARE, mergekit |
| E · Steer, then bake in | Contrast prompts give a direction; add it at inference or fold it into weights. | ActAdd, CAA, RepE, Arditi et al. |
| F · Neuron-level edit | Chosen neurons or rows are trained or set directly. | 128-neuron persona (Qwen3-8B), neuron-explorer, Sparse Feature Circuits |
| G · Agent editing | A general LLM with a sandbox reads raw weights, hypothesises and edits. | Takehi et al., MIT CSAIL, 2026 |
The 2026 headline result is mechanically an E/F edit chosen by a G-type agent: a “loop direction” subtracted from down-projection columns in two layers. That does not make it unimportant. Automating the discovery and the application is the novelty. It does set honest expectations.
What the MIT result established
Solid
- For small networks (up to about 1.5 million weights) a frontier model with a Python sandbox can read function from raw weights and write working policies from scratch. Logs and diffs are public.
- One precise, tiny, gradient-free edit (0.016 %) can remove a failure mode in a small reasoning model. The exact diff is on Hugging Face; an independent re-run reproduced the MATH-500 result (4 samples). The GSM8K and IFEval figures have not been re-run yet.
Contested or weak
- Pure weights-only reading at LLM scale did not work: a 270M sycophancy test failed from weights alone, and the 0.8B repair needed fifty forward passes.
- The capability gain is small once baselines are fair: the base model with thinking off already scores 45.8 %; the fair delta is about +4.3 points, and with thinking off the edit costs about two points (−2.2).
- The editor is a closed frontier model with undisclosed cost per edit. A clean, pre-registered replication with open editors would settle the prompt-leakage objection. That is our first deliverable.
The gaps we take on
- 01Verification: diff → neurons → before/after chat → regression battery in one tool. The critics of the result asked for exactly this.
- 02Open editors: reproduce the experiments with open-weight agent models; publish cost per edit and pass rates at fixed budgets.
- 03Hybrid editing: the agent chooses targets, a null-space solver applies them, the Atlas verifies.
- 04Grafting compiled modules into pretrained models: unsolved, high value, honest about failure.
- 05A benchmark that separates weights-grounded editing from reason-then-write, with compute accounted for.
- 06Governance of editor agents: a reference design for sandboxing an agent that writes other models’ weights.
Who we are reading
Takehi, Teoh, Zhang and Khattab (MIT CSAIL); Schmidhuber, Schlag and Irie on fast weights; Meng, Mitchell, Fang and Gupta on editing and its limits; Lindner and Shaw on compiled transformers; Arditi, Turner and Zou on steering; Charakorn and Sakana AI on text-to-weights; lyra on neuron-explorer; Cammarata, Gogineni and Tigges at Goodfire on clean neurons; Zhuge on Neural Computers; Lin on Neuronpedia; the circuit-tracer and Transluce teams.
The full research review (79 verified arXiv papers, 33 repositories with licences), the architecture, the roadmap and the verification ledger are available to reviewers on request.