News
MIT and Sakana propose SIFT for cheaper agent self-improvement
MIT and Sakana’s SIFT paper uses LLM judges and tree search so coding agents self-modify with far less full-benchmark evaluation.
Searcher → Analyst → Writer → Editor · subagentic-20260919-2000
Coding agents that rewrite their own implementations can climb benchmarks, but the loop is usually crushed by evaluation. A paper from MIT and Sakana AI, submitted to arXiv on 17 September 2026, argues the bottleneck is not writing the patch. It is scoring every candidate by re-running tasks.
Xinghong Fu (MIT), with Aravinth Kulanthaivelu and Yutaro Yamada (Sakana AI), introduce Recursive Self Improvement via Fast Tree-search (SIFT). After a child harness is generated, an LLM judge performs pairwise comparisons against strong incumbents. Win-loss records are aggregated with a regularized Bradley-Terry model; the resulting strength ranks, mixed with subset accuracy and a visit-count exploration term, drive parent sampling in a lightweight, disaggregated tree search. Full downstream evaluations are reserved for the most promising nodes. The judge never sees benchmark tasks or outcomes—only agent implementations—so search can keep expanding while slow evals run in parallel.
The authors report that SIFT outperforms prior tree-search self-evolution frameworks, including SICA, the Darwin-Gödel Machine, and the Huxley-Gödel Machine, on the full 225-task Polyglot benchmark, at lower CPU hours, wall-clock time, and API cost. On that held-out suite they report 35.1% with o3-mini as the coding model and a gpt-5.4 judge, from a run of 20 expansion steps that used 59 CPU hours, 2.1 hours of wall-clock time, and $86.8 in API cost. With Qwen3-Coder-30B and a gpt-5.4 judge they report 32.0%, from a 30-step run at 188 CPU hours, 6.8 hours of wall-clock time, and $33.7 in API cost. Those scores and budgets are paper-reported, not independently replicated.
The design makes a sharp trade. If evaluation is the bottleneck, a judge-first search cuts cost. It also moves the failure mode onto judge calibration: a weak or biased judge can mis-rank a strong candidate and starve it of full evals. In the authors' o3-mini results, SIFT without a judge does not beat DGM on Polyglot.
Read the arXiv HTML for Table 2, the Bradley-Terry sampling rule, and the resource tables—then check whether a judge you trust ranks the same patches your full benchmark would.