NVIDIA open-sources SkillEvaluator; Hermes scans skill installs before confirm

NVIDIA’s August 19, 2026 technical blog presents SkillEvaluator as open source, with the first catalog-wide numbers on how verified agent skills change live trajectories. The same day, Nous Research said Hermes now runs that evaluator on skill installs — advisory scans before the user confirms.

That pairing is the story. Skills are becoming the packaging layer for agents: instructions, examples, and tool guidance bundled so a model spends fewer steps finding the right product surface. If the skill is the unit of distribution, a SKILL.md dump is not enough. Teams need performance evidence and an install-time look at PII, secrets, licensing, and smuggling.

A measured catalog

NVIDIA's technical blog describes SkillEvaluator as an open-source layer for measuring how a skill affects agent performance. It combines static checks with paired live runs of the same task — once with the skill installed, once without.

The first published snapshot covers more than 300 verified skills across over 30 NVIDIA products. Each skill ran on two independent harnesses. Skill Lift is the with-skill score minus the without-skill score, reported in points, not percent change.

NVIDIA verified skills are packaged, signed capability descriptors. They tell an agent what a product does, when to invoke it, and how to call it. The company is explicit that the “verified” part is the measurement that determines a skill is ready.

The same skills ship as plugins for Claude Code, Codex, and Cursor. They are also listed on Skills.sh, ClawHub, and Hermes Hub.

Three tiers, then a sandbox

Before a skill is published, it can pass through three tiers. Each answers a different question, and each can run on its own.

Tier 1 — Safety and structure. Static checks for schema and frontmatter validation, quality scoring, security scanning for prompt injection and data exfiltration, secret and PII detection, license checks, and script linting.

Tier 2 — Distinctiveness. Embedding similarity to catch duplicated guidance inside a single skill and overlapping coverage across the catalog.

Tier 3 — Live evaluation. An agent runs generated tasks in an isolated sandbox, with the skill and without it. The difference is Skill Lift.

Live runs use Harbor, an open-source framework for repeatable isolated agent evaluations. SkillEvaluator turns cases into Harbor tasks, runs the sandboxes, collects results, and computes lift. Within each harness the only experimental variable is whether the skill is installed. NVIDIA repeats that comparison across two harnesses.

A typical workflow from the post looks like this:

skillevaluator create-eval-dataset ./my-skill --full
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker

--full builds explicit, implicit, contextual, and negative cases into evals/evals.json. After review, the Tier 3 command converts those cases into a Harbor bundle, grades both runs, and reports Skill Lift.

What the August snapshot shows

The figures in the post come from an August 12, 2026 snapshot of benchmarks.json. Scores are macro-averaged so each published skill–harness pair gets equal weight. NVIDIA says the catalog is evaluated continuously.

Without a skill, average baselines sat between 39 and 46 out of 100 on four of the five dimensions: Correctness 46, Discoverability 42, Effectiveness 39, and Efficiency 43. Security was already 97. For Security, the stated objective was mainly to verify that installing a skill did not introduce a regression.

Those dimensions are not interchangeable. Correctness asks whether the final answer is right. Discoverability asks whether the right skill loads when it is relevant and stays unloaded when it is not. Effectiveness asks whether the agent reached the user’s goal and followed the expected workflow. Efficiency asks whether it got there without wasted steps or redundant tool calls. Security covers unsafe operations, secret leakage, and unauthorized access.

NVIDIA flags a scoring quirk that matters when you read the baselines. Discoverability and Efficiency also measure how a skill is used — whether the agent finds it, reads it before acting, and avoids unnecessary steps. Without the skill, those actions are unavailable, so those baselines are not expected to be zero. An agent can still earn credit for productive tool use, clean execution, and correctly leaving a skill unloaded on unrelated tasks. That is why those averages sit around 42 and 43 rather than at the floor.

On the with-skill side, the post says verified skills improved agent performance across both evaluated harnesses — Codex and Claude Code — with the largest gains in Correctness, Discoverability, Effectiveness, and Efficiency. Security was already high at baseline. NVIDIA reports Skill Lift in points as a with-skill minus without-skill delta across skill–harness pairs; this piece does not restate a single catalog-wide headline lift per dimension.

Limitations are written down. Of the skills with published results, 85% ran one attempt per task and 15% ran two. Live agent runs vary between trials, so individual skill scores vary. Catalog-wide averages aggregate thousands of trials, but the post does not report confidence intervals.

Hermes checks the package before confirm

On August 19, Nous Research said Hermes now leverages NVIDIA’s SkillEvaluator on skill installs. The scan looks for PII, leaked secrets, Unicode smuggling, licensing, and security issues before the user confirms.

Nous pointed the scanner at its own bundled skills first and used what it found to improve 11 of them.

That is the other half of the packaging story. Skill Lift is a catalog metric from controlled with/without runs. The Hermes check is an advisory scan of what a user is about to add. A signed descriptor can still carry secrets, PII, smuggling, or a license problem. Scanning before confirm is how a hub surfaces those as findings instead of documentation footnotes.

The checks line up with SkillEvaluator’s Tier 1 surface: secret and PII detection, license checks, and security scanning. Hermes is applying that surface at the moment a user is about to add a hub skill, not only at the moment NVIDIA decides a verified skill is ready to publish.

Skills as a scored package

NVIDIA’s framing is that agents are only as effective as the context they receive. Even with capable models and documented libraries, agents still burn steps finding tools, spend tokens on dead ends, or stall on specialized tasks. Skills are the package meant to shorten the path from intent to solution. SkillEvaluator is the claim that those packages can be scored, not just shipped.

Hermes treating SkillEvaluator as an install-time check is the claim that a hub should not wait for a human to read every file. Performance evidence and a confirm-time scan are different jobs. One tells you whether the skill helps. The other flags issues before you confirm.

If you maintain skills, the practical sequence is the one NVIDIA documented: generate the eval dataset, review the cases, run the live comparison in an isolated sandbox, and keep the static safety pass in the loop. Then treat install-time findings the way Nous did — fix what the scanner catches before users confirm.

Next step: Read NVIDIA’s SkillEvaluator technical blog for the tier definitions, Harbor workflow, and August 12 snapshot tables, then read the Nous Research announcement for how Hermes surfaces those checks before confirm.