Continual-ARC: shipped harness notes did not memorize recurring tasks

On 11 November 2026, Dan Lenton published Unify’s results on Continual-ARC, a benchmark the lab built so a harness that keeps notes and a model that updates its weights face the same stream of work. These are the lab’s own runs, not an independent leaderboard. As shipped, OpenClaw, Hermes Agent, and Prime Agent saved nothing measurable on that stream when driven by GPT-6 Luna. Told what to store, they did save a measurable share. On Qwen3.5-9B, the student used for the notes-versus-weights comparison, notes as shipped learned nothing measurable. Weights trained to imitate a teacher saved spend on puzzles the model had met and on puzzles it had not.

The lab built the stream because most continual-learning benchmarks for agents serve each task once. A remembered answer then cannot be told apart from a learned skill, and few checks ask whether the agent later gets cheaper on tasks it never saw. Continual-ARC serves the same puzzles again with fresh inputs. A cached grid does not fit. What is worth carrying is the rule.

On the ARC-AGI-1 track, the stream draws 25 tasks from the 400 training puzzles and serves 200 instances. Four puzzles are active at a time, each for about three visits, then a rest of about thirty, until each of the 25 has had eight visits. The last new puzzle arrives in round 97 of 200. The puzzle’s name is shown on every visit, so failing to recall a solution is not the same as failing to recognize the task.

Each visit has a feedback budget of sixteen. One more worked example costs one. A wrong answer costs one. A visit that never gets the grid right costs all sixteen, so solving is always cheaper than failing. A raw score would mostly measure the model. The lab instead compares each learner with itself reset, everything it kept wiped before every visit, and reports the share of that reset’s spend it no longer pays. Zero is no change. One is the whole test for free. A gain counts only when a task-resampling interval, corrected across the main comparison, leaves out zero.

After the stream the learner is frozen and given two tests of 100 visits. The seen test is fresh inputs of the 25 puzzles it met. The held-out test is 100 ARC-AGI-1 evaluation puzzles the stream never served. Memory should get cheaper on the first. Getting better at ARC should get cheaper on both. The same design also runs on ARC-AGI-2 and ARC-AGI-3. The harness numbers below are from the ARC-AGI-1 experiments.

Shipped notes did not come back

Prime Agent, Hermes Agent, and OpenClaw were run on GPT-6 Luna at three reasoning efforts, in case the result was Prime Agent’s alone. As shipped, none saved a measurable share on the stream. At high effort, not one of their solves after round 100 was free. By then every puzzle had been met, and a perfect memory would have spent nothing on a solved visit.

The logs show why the notes did not pay off. Prime Agent never marked a lesson global in 600 sessions, and every session opened on "No saved harness entries yet." OpenClaw, asked to save, wrote 47 to 84 notes per run and read one named for the puzzle on at most 1 percent of the returning visits its log kept.

Told the lab’s program-library procedure — keep one checked Python file per puzzle and run it first when that puzzle returns — every harness saved a measurable share at every effort, 0.17 to 0.62. That is a procedure the harness was told to follow. It is not evidence that notes, left alone, had learned the rule.

Weights moved the unseen puzzles

The notes-versus-weights comparison uses one open student, Qwen3.5-9B, and one teacher, GLM-5.3 Flash. Left to itself, Qwen’s harness saved nothing measurable, on puzzles it had met or on new ones. Neither the lab’s guidance nor GLM’s notes alone saved more than chance on the seen test. With both, Qwen learned 0.29 on the stream and saved 0.34 on the seen test. On the held-out test, no version that keeps notes saved more than 0.03.

Trained between rounds to imitate GLM’s moves, Qwen’s weights saved 0.57 of the untrained model’s spend on the seen test, from 84 percent of the budget to 36 percent, and 0.29 on the 100 new puzzles. The trained model solved about 49 of those 100, against 19 for the untrained model in the harness and 31 with no harness at all. A page of playing rules, which the teacher read and the student did not, accounts for part of the new-puzzle gain. Told that page alone, the untrained model saved 0.09 there. Training saved 0.19 more than the page, and on puzzles already met the weights kept far more than the page could, 0.57 against 0.16.

The Continual-ARC site and the paper abstract, arXiv:2611.14505, submitted 10 November 2026, state the same design claim for the methods they report: persistence in the harness generally does not improve memorization unless the harness is given explicit expert guidance, and no harness-based method gained on unseen tasks, while weight-based learning did. The abstract does not name OpenClaw, Hermes Agent, or Prime Agent. Those three scores are the lab’s GPT-6 Luna runs, written up in the 11 November post and linked from the benchmark site’s replay. Lenton wrote the same day that the three harnesses were not able to memorize anything of value and were not able to generalize to new tasks, and that training the 9B weights did both.

What the lab will not claim

The post lists the limits. The stream comes from ARC-AGI-1’s public training set, which every model developer knows, and a model may know some of these rules already. Notes and weights are compared on one open student, one teacher, and one seed. None of the models is the strongest available, and a stronger one might keep notes that answer a returning puzzle. Office work also has no task ids to file a program by. From this stream the lab still offers three harness lessons: store liberally, including examples already paid for, and prune later; store functions, not only notes, checked against those examples; and bring stored work back without being asked, which OpenClaw did not. For deployment, the post’s conclusion is to train the model that drives the harness as well as give it a memory. Next, the lab says, is Drift, a stream in which a puzzle’s rule changes partway through and what a learner keeps can go stale.

Read the 11 November post for the tables, then open a GPT-6 Luna run in the Continual-ARC replay and the leaderboard rather than stopping at the summary. The arXiv abstract states the scoring rules in short form. If you have a learner to test, the site explains how submissions are checked on tasks the lab has not published.

Sources

• Why we built Continual-ARC unify.ai
• Continual-ARC continual-arc.unify.ai
• Continual-ARC: Measuring Skill Acquisition over a Working Lifetime arxiv.org
• Daniel Lenton on Continual-ARC, 11 November 2026 x.com