subagentic.ai RSS X YouTube GitHub
AI-generated illustration

News

GitHub Copilot Evals is in public preview

GitHub Copilot Evals is in public preview so teams can retest skills, custom agents, and MCP tools after a setup change.

github-copilotevalsmcpagent-skills

GitHub Copilot Evals is in public preview. The November 17, 2026 changelog says you can use it to optimize Copilot for yourself: as prompts, models, skills, and tools change, evals help you check whether the skills, agents, and MCP tools in your setup are still doing what you want.

You create an eval, inspect how those pieces behave, and use the results to decide what to improve.

GitHub defines an eval as a repeatable test for AI behavior. You give an agent a realistic task, describe what a good result looks like, and check both the result and how the agent got there. Output can change from one run to the next, so one good answer does not tell you whether a customization works well for you. After you change a skill, agent, or MCP in your Copilot setup, you rerun the same eval to check the behavior you care about and catch regressions.

You can ask Copilot in your own words or use the /eval command. The changelog lists four things that command, or a plain-language ask, can do:

  • Create an eval for a skill, custom agent, or MCP tool in your project.
  • Run it and see which checks passed and failed. You approve each run before it starts.
  • Find out why a check failed and decide what to change in your Copilot setup.
  • Compare runs from before and after a customization change to see whether it helped.

The evaluation engine is Vally, which GitHub describes as an extensible local eval tool. How Vally scores a check, and how far that extensibility goes, is not stated in the changelog.

Install GitHub Copilot Evals where you already use Copilot:

  • In Visual Studio Code, install the GitHub Copilot Evals extension from the Visual Studio Marketplace.
  • In Copilot CLI, install the GitHub Copilot Evals plugin.
  • In the GitHub Copilot app, open Customize, add the github/evals marketplace, and install the GitHub Copilot Evals plugin.

The preview is available on Copilot Individual, Copilot Business, and Copilot Enterprise plans.

If a skill, custom agent, or MCP tool in your setup has changed, install Evals in the client you already use, write one eval for a behavior you already care about, approve a run, and keep that eval to rerun after the next change. Further reading is the GitHub Copilot Evals documentation linked from the changelog. Feedback goes through an issue on the evals repository or the Copilot conversations category in GitHub Community.

Sources