How do I debug failing eval tests?

You can enable verbose output by setting E2E_VERBOSITY=verbose in your terminal, which displays the full conversation between the LLM and your MCP tools.

Can I run individual test files?

Yes, the suite supports standard Jest commands, allowing you to run specific unit tests or eval files by passing the file path to the test command.

How do I mock APIs in these tests?

You can use the USE_MOCK_API=true environment variable and the configureApiClient helper to set up a mock environment before running your tests.

What are MCP eval tests?

Eval tests use an LLM agent to interact with your tools end-to-end, simulating real-world usage scenarios to ensure workflow completion and reliability.

Which SDK does this skill support?

This skill is specifically designed for the @umbraco-cms/mcp-server-sdk, providing patterns for its testing and eval modules.

MCP Testing Suite

Name: MCP Testing Suite
Author: umbraco

byumbraco

Security & Testing

Provides standardized patterns and utilities for unit testing and LLM-driven evaluation of Model Context Protocol (MCP) tools.

About

The MCP Testing Suite skill streamlines the quality assurance process for Model Context Protocol (MCP) servers using the Umbraco MCP SDK. It offers pre-defined patterns for traditional unit testing—including request handler mocking and environment setup—alongside advanced 'eval' tests that use LLM agents to validate end-to-end tool workflows. This skill is ideal for developers who need to ensure their MCP tools handle complex logic correctly and maintain reliability through both static assertions and dynamic scenario simulations.

Key Features

0 GitHub stars
Mocking utilities for MCP request contexts and API clients
Ready-to-use configuration for Jest and environment setups
LLM-driven eval testing for end-to-end scenario validation
Verbose debugging support for complex tool interactions
Standardized unit testing patterns for MCP tool handlers

Use Cases

Writing unit tests for individual MCP tools with mocked API responses
Conducting end-to-end workflow validation using LLM agents to simulate user tasks
Debugging MCP server failures through verbose logging and specialized test environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add umbraco/umbraco-mcp-base mcp-testing

For use in Claude.ai and ChatGPT

Download Skill

GitHub