Skip to content
worth noting Coding

GitHub Copilot Evals tool for testing AI agents enters public preview

clearly official source

GitHub has released GitHub Copilot Evals into public preview — a tool for repeatable testing of AI agents, skills, and MCP tools in Copilot, with the Vally evaluation engine, available for Individual, Business, and Enterprise plans.

GitHub has released the GitHub Copilot Evals tool into public preview. The tool is used to create repeatable tests (evals) for AI behavior within Copilot setups — specifically for agents, skills, and MCP tools. According to the documentation, the user gives the agent a realistic task, describes what a good result should look like, and the eval then checks both the result itself and the process by which the agent arrived at it.

The reason for this is that AI output can vary between individual runs, so a single successful result doesn't indicate whether the setup works reliably. After changing a skill, agent, or MCP tool in Copilot, the same eval can be run again to verify that the monitored behavior has been preserved, or to catch regressions.

The tool can be controlled directly within Copilot using natural language or the /eval command, or installed as a plugin from the github/evals marketplace. It uses Vally as its evaluation engine, described as an extensible local eval tool. GitHub Copilot Evals is available in public preview for the Copilot Individual, Copilot Business, and Copilot Enterprise plans.

What changed

Why it matters

Developers and companies deploying custom agents, skills, and MCP tools in Copilot have so far had to rely on random trial and error to find out whether a change to a setup improved or worsened AI behavior. Repeatable evals allow systematic verification of regressions after every change to a prompt, model, or tool, which reduces the risk of a broken or less reliable configuration slipping into the production workflow.

Two audiences, two different impacts

What this means

01

For individuals

GitHub Copilot users can now verify whether, after changing a prompt, model, skill, or MCP tool, they still get equally good results, and catch regressions before noticing them in everyday work.

What to do Try creating an eval for your own Copilot setup (skill, agent, or MCP tool) using the /eval command or by installing the GitHub Copilot Evals plugin.
More practical updates →
02

For a business

Companies using GitHub Copilot in the Business or Enterprise tier can incorporate the tool into their development process as a repeatable test for checking regressions when modifying agents, skills, or MCP tools, which reduces the risk of uncontrolled changes in AI behavior across the team.

Development
What to decide Consider deploying GitHub Copilot Evals to test custom skills, agents, and MCP tools used in a Copilot Business/Enterprise environment.
More business impacts →
AI agents evaluation GitHub GitHub Copilot GitHub Copilot Evals MCP

Check the original

Event sources

clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
GitHub Copilot Changelog primary source · first detected GitHub Copilot Evals is now in public preview