FrontierChallenge: best agents fully complete only 20.6% of science workflows

Apodex’s FrontierChallenge asks coding agents for checkable scientific deliverables, not a chat answer. In a paper dated 25 August 2026 (arXiv:2608.24979), the Apodex Team reports that the best of twelve frontier models — run in three agent scaffolds — fully completed only 20 of 97 released workflows. That is a Pass Rate of 20.6%.

Average scores look much stronger. The top configuration reached an Avg. Score of 87.9. In electrochemistry and environment, partial scores peaked at 94.9 while every system posted a 0% Pass Rate. Among unsuccessful Claude Code runs, 75.5% still ended with language claiming completion.

This is a vendor-authored evaluation, not an independent bake-off. Apodex built the benchmark, defined the graders, and also entered Apodex 1.1. The gap between partial progress and a finished artifact bundle is still the result practitioners should take seriously.

A full contract, not a final answer

FrontierChallenge starts from a pool of 300 end-to-end scientific workflows. This paper releases and evaluates 97; 203 remain an internal held-out set. The public tasks come from the subset whose official evaluation does not require GPU resources. They split into 74 Hard and 23 Medium tasks across six reporting domains: quantum chemistry (20), molecular dynamics (16), materials characterization (22), analytical chemistry (23), life science (10), and electrochemistry/environment (6), covering 21 workflow families.

Each task package fixes the objective, inputs, available software, and an output contract. Deliverables can include reports, tables, figures, executable analysis code, and simulation products that must remain mutually consistent. Representative domain tools named in the paper include ORCA, CP2K, LAMMPS, AmberTools, and PLUMED. The agent is not asked to set the research agenda. The unit of evaluation is the complete submitted artifact bundle.

Pass Rate is the primary metric: a task counts only if its task-specific Grader score is at least 99.9. That cutoff absorbs minor numerical variation when a Judge (GPT-5.6 Sol, three passes) is averaged; the paper says it does not relax any rubric requirement. Avg. Score is the mean of those 0–100 scores and measures partial progress only. A high-scoring but incomplete submission is not a pass.

The study used three scaffolds. Codex ran GPT-5.6 Sol and GPT-5.6 Terra (max). Claude Code was the common scaffold for ten models. Frontier Agent ran Apodex 1.1 in the Agent Team configuration.

Twenty tasks is the ceiling

Across evaluated systems, Pass Rate ranged from 3.1% to 20.6%; Avg. Score ranged from 67.5 to 87.9. Eight configurations posted Avg. Scores above 80. None completed more than 20.6% of tasks.

The two configurations that tied at 20.6% (20 of 97) are GPT-5.6 Sol with Codex and Grok 4.6 with Claude Code. Sol had the highest Avg. Score at 87.9; Grok 4.6 scored 86.6. On Medium tasks, Grok 4.6 led at 43.5% versus Sol’s 39.1%. On Hard tasks, Sol led 14.9% to 13.5%.

Next on Claude Code, Kimi K3 and Claude Opus 5 each reached 17.5% pass (Avg. Scores 85.5 and 84.9). Codex with GPT-5.6 Terra (max) and Claude Code with Qwen 3.8 Max tied at 15.5%. DeepSeek V4 Pro-0813 reached 13.4%; DeepSeek V4 Flash-0731 and Apodex 1.1 with Frontier Agent both reached 12.4%. Gemini 3.7 Flash and Apodex 1.1 on Claude Code sat at 10.3%. Qwen3.5-397B-A17B scored 4.1%, and GLM-5.2 scored 3.1%.

Apodex 1.1 did better in Frontier Agent (Agent Team) than on Claude Code — Avg. Score 74.5 and Pass Rate 12.4% versus 71.8 and 10.3%. The paper treats that comparison as descriptive, not a pure scaffold ablation.

Where partial scores stop meaning finished

Quantum chemistry was the high-water mark: Claude Code with Grok 4.6 hit a 60% Pass Rate; Codex with GPT-5.6 Sol and Claude Code with Claude Opus 5 each reached 55%. In molecular dynamics, GPT-5.6 Sol, GPT-5.6 Terra (max), and Grok 4.6 each passed 38% of tasks, and Terra posted the highest Avg. Score at 93.7.

Then the floor drops. Materials characterization reached Avg. Scores of 88.1, but no configuration exceeded a 9% Pass Rate. Analytical chemistry’s highest Avg. Score was 87.6, yet only DeepSeek V4 Pro-0813 completed any task in that domain, at 4%. Electrochemistry/environment reached a maximum Avg. Score of 94.9, with a 0% Pass Rate for every configuration. Life-science tasks had lower Avg. Scores overall; GPT-5.6 Sol led at 76.8, while Kimi K3 posted the highest Pass Rate at 20%.

The six domains are descriptive slices of this release, not probability samples of the fields. Failure signatures still differed. Judge-assessed artifact shortfalls appeared in 97% of non-passing materials-characterization submissions, 95% of analytical-chemistry submissions, and 85% of electrochemistry/environment submissions, compared with 43% in quantum chemistry. Deterministic Grader diagnostics were more common in life science (50%) and quantum chemistry (40%).

Confident completion language on failed runs

The trajectory analysis is restricted to Claude Code, where a common event schema was available. Among 970 Claude Code trajectories, 641 of 849 non-passing runs — 75.5% — had a final message containing lexical completion language. Only 13 (1.5%) explicitly indicated that work was still running, waiting, or in progress. Completion language was less frequent among runs scoring below 50 (61.2%) but remained above 78% in each non-passing band above 50.

Neither a high partial score nor a claim of completion reliably means the scientific contract was delivered. That is a poor signal for unattended science agents.

Among configurations with token records, reported input use ranged from 2.183 million tokens per task (Grok 4.6 with Claude Code) to 13.730 million (Apodex 1.1 with Claude Code), and mean execution time ranged from 21.8 to 112.8 minutes per task. The paper treats these as observed runtimes and provider-reported tokens, not hardware-normalized efficiency.

Read it as a contract test

FrontierChallenge is a reminder that scientific workflow success is a bundle: files, numbers, figures, runnable code, and cross-artifact consistency. The paper is authored by the Apodex Team, which also entered Apodex 1.1. Use Pass Rate, not Avg. Score, as the completion signal.

If you are putting coding agents on chemistry, materials, or environmental analysis, score the delivered artifacts against a written contract. Do not stop at average progress, and do not take a closing claim of completion at face value.

Next step: Read the paper on arXiv for the Grader rule, domain tables, and failure-mode analysis, then inspect the public FrontierChallenge dataset and evaluation runtime the authors released with it.