MIT's SIFT Framework Cuts Coding Agent Evaluation Costs with LLM as Judge

Researchers at MIT and Sakana AI have developed a framework called Recursive Self-Improvement via Fast Tree Search (SIFT) to reduce the cost of evaluating modifications to coding agents. The traditional approach to improving coding agents involves testing each candidate change, which can be time-consuming and expensive. However, SIFT uses an LLM as judge to compare candidate implementations pairwise, reducing the need for extensive benchmark runs.

The main challenge in evaluating coding agent modifications is the poor signal-to-cost trade-off of existing evaluation methods. Each candidate change needs testing, but running a full test suite on every modification can consume thousands of CPU hours and cost thousands of dollars. SIFT addresses this issue by using an LLM to compare two versions of an agent before committing to expensive benchmark runs.

SIFT builds upon the Darwin Gödel Machine (DGM) framework, which maintains an archive of agents and checks candidates on increasingly large task sets. However, DGM’s approach can be slow due to its reliance on sequential evaluation. In contrast, SIFT uses a more efficient search strategy that allows it to explore multiple branches in parallel.

The central idea behind SIFT is to use the LLM as judge to compare candidate implementations pairwise. This involves running each new agent on four coding tasks and then comparing it with up to 10 high-ranking agents already in its archive using the Bradley-Terry model, a statistical method for estimating relative strength from wins and losses.

SIFT combines the pairwise choices into a ranking that helps decide where testing and further development are worth spending time. The new candidate is then placed into a priority queue for the larger, time-consuming evaluation. Its priority combines the judge’s ranking with its benchmark-accuracy ranking.

The workflow avoids making each patch wait for the previous one’s full evaluation. Patch generation, judging, and benchmark evaluation can proceed in parallel, similar to a CI pipeline where work continues while longer tests run in the background. This allows promising candidates to become the parents of new patches before their expensive evaluations finish.

SIFT was tested on various benchmarks, including Polyglot, TerminalBench 2.1, and SWE-bench Verified subset. The results showed that SIFT delivered better accuracy with less compute using both open-weight Qwen3-Coder-30B and closed o3-mini models. With Qwen3-Coder-30B, it slightly outscored HGM while using about a third fewer CPU hours.

On Polyglot, one SIFT run reached 35.1% accuracy in under five hours of wall-clock time, using 42 CPU hours and about $150 in API credits. This is a significant improvement over DGM’s performance on the same benchmark. Additionally, SIFT showed that it can handle large task sets efficiently by achieving better results with less compute.

SIFT was also tested without the LLM judge to see whether it made a difference. The results showed that the judge helped drive the gain in accuracy and efficiency. This suggests that using an LLM as judge is crucial for SIFT’s success.

The researchers found that a cheaper judge retained much of the global ranking signal on TerminalBench, although the stronger model was more reliable among the best candidates. They suggest that a tiered setup could work, with the cheaper judge for most comparisons and a stronger one near the top of the ranking.

SIFT shows how the self-improvement loop itself can become more efficient by combining cheaper evaluation signals with asynchronous search. This allows teams to explore more candidate improvements without scaling evaluation costs at the same rate. However, downstream benchmarks are still needed to confirm that a change actually improved the agent.

The framework is built on top of the DGM harness and adds the pairwise judge and Bradley-Terry ranking. It makes the search asynchronous so candidates can be expanded while others are still being evaluated. SIFT was only tested on coding agents but its basic approach could apply to other self-improving agents where there’s a clear way to measure whether a change worked.

The first step in applying SIFT is to define a small, cheap evaluation that can quickly reject obviously bad modifications. For an enterprise agent, this could be a small set of representative internal tasks that catches broken tools, regressions or changes that violate basic requirements. The second piece is an LLM judge that compares candidate implementations pairwise.

SIFT’s efficiency comes from its ability to explore multiple branches in parallel while previous candidates are still being tested. This allows the search tree to keep growing instead of waiting for every evaluation to finish. SIFT shows how AI tools can be used more effectively by combining cheaper evaluation signals with asynchronous search, making it a valuable addition to any business’s toolkit.