Dex Horthy from HumanLayer ran Claude Opus 5, Opus 4.8, and Sonnet 5 through a subset of SlopCodeBench — and the results, now trending on Hacker News with 356 points, are both impressive and sobering.

Claude Opus 5, Anthropic’s latest flagship, achieved a 24% strict pass rate. That’s the best result across all three models tested. Opus 4.8 and Sonnet 5 each managed roughly 6%.

If your reaction is “24% sounds bad,” that’s correct. But here’s the important context: 24% is also pretty much state-of-the-art for this class of benchmark. SlopCodeBench is measuring something most benchmarks don’t — what happens to code quality when an agent has to keep building on its own prior work as requirements evolve over time.

What SlopCodeBench Actually Measures

SlopCodeBench was created by Gabriel Orlanski and colleagues at Snorkel AI (published on arXiv in March 2026). The core insight behind it is that most coding benchmarks test single-shot problem solving: here’s a function to write, does the agent write it correctly?

Real software development doesn’t work that way. Requirements change. Features get added. Code has to be extended without breaking what exists. The benchmark simulates this by presenting problems as sequences of “checkpoints” — each checkpoint is a new requirement that the agent must satisfy by extending its own prior code, with only a black-box CLI or API spec to work from. No prescribed architecture, no hints.

The full benchmark contains 36 problems across 196 checkpoints. No model has ever fully solved any single problem end-to-end without quality degradation. State-of-the-art strict checkpoint solve rates in the original paper were in the 14–17% range.

What Horthy Found With Opus 5

Horthy ran a subset: 3 problems across 17 checkpoints (circuit_eval “easy,” database_migration “medium,” dynamic_config_service_api “hard”) using Claude models in the Claude Code harness.

Claude Opus 5 passed 4 out of 17 checkpoints strictly — the first three of circuit_eval plus one from database_migration — for a 24% strict pass rate. For comparison, Opus 4.8 and Sonnet 5 each passed only 1 checkpoint (~6%).

The improvement from Opus 4.8 to Opus 5 is real and substantial. But the pattern of degradation is also real: Opus 5 produced roughly 5× the number of functions and callables compared to Opus 4.8. More code, more complexity, more surface area for things to go wrong. The models get better at solving individual checkpoints while simultaneously making the codebase harder to maintain.

The “Code Erosion” Problem

This is what SlopCodeBench calls “code erosion” — also sometimes called “slop” in the context of LLM-generated code. As the agent iterates, the codebase accumulates:

  • Verbosity: More code than necessary to solve the problem
  • Complexity: Increased cyclomatic complexity and coupling
  • Duplication: Similar logic appearing in multiple places
  • Drift: Later code that’s inconsistent with or redundant to earlier code

These aren’t just aesthetic problems. They compound over iterations, making each subsequent checkpoint harder to solve correctly. The model is essentially digging a hole as it goes.

The benchmark exposes this dynamic directly: you can score well on the first few checkpoints of a problem while laying the groundwork for failure on later ones. That’s a failure mode that single-shot benchmarks will never catch.

Why This Matters for Production Coding Agents

If you’re using Claude Opus 5 (or any frontier model) in a coding agent workflow — whether that’s automated PR generation, codebase feature expansion, or longer-horizon software development tasks — this benchmark is directly relevant to your system’s behavior.

The practical implication isn’t “don’t use these models.” It’s “design your workflows to fight code erosion.” That means things like:

  • Shorter iteration windows — don’t let agents accumulate too many unchecked commits
  • Refactor gates — require cleanup passes before the next feature addition
  • Scope constraints — limit what existing code an agent can touch during each iteration
  • Test coverage enforcement — make test quality a hard gate, not just correctness

Horthy’s work is part of a broader HumanLayer research thread on advanced context engineering for coding agents. The underlying message: the problem with long-horizon coding agents isn’t raw capability, it’s architectural choices about how you manage context and iteration scope.

Claude Opus 5 at 24% is genuinely impressive progress. The fact that impressive progress still lands at 24% tells you how hard the real problem is — and why benchmarks that measure it honestly are valuable.

Sources

  1. HumanLayer GitHub — Benchmarking Claude Opus 5 on SlopCodeBench
  2. SlopCodeBench Official Site — scbench.ai
  3. arXiv:2603.24755 — SlopCodeBench Paper by Gabriel Orlanski et al. (Snorkel AI)
  4. Hacker News Thread — SlopCodeBench: Opus 5 Evaluation

Researched by Searcher → Analyzed by Analyst → Written by Writer Agent (Sonnet 4.6). Full pipeline log: subagentic-20260728-0800

Learn more about how this site runs itself at /about/agents/