Can I run evals against a mock API?

Yes, the skill automatically detects if mock handlers exist in your project and configures the eval environment to use them if available.

What happens if an eval test fails?

Since evals are probabilistic, failures are often fixed by iterating on the prompt instructions or tool descriptions to make them more explicit for the LLM.

What is the difference between integration tests and eval tests?

Integration tests are deterministic and test tool handlers directly against an API. Eval tests are probabilistic; they use the Claude Agent SDK to test if an AI can successfully navigate workflows using your tools.

Why must I run 'npm run build' before running evals?

Evals test the MCP server by executing the compiled code in 'dist/index.js' to ensure the tests accurately reflect the behavior of the deployed toolset.

Do I need an API key to run these evals?

Yes, eval tests require an LLM API key (either through a Claude Code subscription or an ANTHROPIC_API_KEY) because they involve real LLM calls to test the agent logic.

Build Evals for MCP

Name: Build Evals for MCP
Author: umbraco

byumbraco

•

Seguridad y Pruebas

Generates LLM-powered evaluation tests for MCP tool collections to validate AI agent performance and workflow reliability.

The build-evals skill automates the creation of sophisticated LLM evaluation tests for Model Context Protocol (MCP) servers. By analyzing tool definitions and discovery manifests, it constructs realistic workflow scenarios—such as CRUD lifecycles and hierarchical navigation—using the Claude Agent SDK. This allows developers to move beyond basic unit testing and verify how effectively an AI agent can actually use their tools to solve multi-step problems in a probabilistic environment, ensuring that prompts and tool descriptions are optimized for real-world AI interactions.

Características Principales

01Support for both mock and live API testing environments

02Intelligent workflow scenario design based on tool metadata and capabilities

03Standardized setup for centralized evaluation and Jest configuration

04Seamless integration with the Claude Agent SDK for probabilistic testing

051 GitHub stars

06Automated generation of Jest-based eval test files for MCP collections

Casos de Uso

01Testing the robustness of tool descriptions and prompts against real-world LLM usage

02Regression testing MCP servers to ensure updates don't break AI agent reasoning or tool selection

03Verifying that an AI agent can correctly sequence multiple MCP tools to complete a complex task

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add umbraco/umbraco-mcp-base build-evals

For use in Claude.ai and ChatGPT

Download Skill