What benchmarks are supported out of the box?

It provides templates and guidance for SWE-bench, CyberGym, and MCPToolBench++.

Why is the {workdir} placeholder required in the config?

The mcpbr runner uses the {workdir} placeholder to dynamically inject the path of the specific task repository at runtime, which is necessary for the MCP server to access the codebase being tested.

How does it handle API keys and sensitive data?

The skill follows best practices by using environment variable placeholders like ${API_KEY} in the YAML files, ensuring your secrets aren't hardcoded.

Can I use this for SWE-bench evaluations?

Absolutely. It provides specific configuration templates for both SWE-bench Lite and SWE-bench Verified datasets.

Does this skill support Python-based MCP servers?

Yes, it includes specific support and validation for Python environments using uvx or direct python/python3 commands.

MCP Benchmark Configurator

Name: MCP Benchmark Configurator
Author: greynewell

bygreynewell

•

Security & Testing

Generates and validates optimized YAML configurations for the Model Context Protocol (MCP) Benchmark Runner.

About

The mcpbr-config skill simplifies the process of benchmarking MCP servers by ensuring configuration files are syntactically correct and follow the strict requirements of the Model Context Protocol Benchmark Runner. It automates the creation of complex YAML files, handles essential requirements like the {workdir} placeholder, and validates execution commands for Node.js and Python environments. This skill is essential for developers looking to evaluate server performance against standardized datasets like SWE-bench, CyberGym, and MCPToolBench++ without manual configuration errors.

Key Features

Automated generation of valid mcpbr YAML configuration templates
Validation of critical {workdir} placeholders to prevent runtime benchmarking errors
20 GitHub stars
Pre-configured templates for SWE-bench, CyberGym, and MCPToolBench++
Support for simplified model aliases and secure environment variable handling
Verification of system executables including npx, uvx, and python

Use Cases

Validating configuration files before running large-scale benchmark jobs to save time and API costs
Setting up standardized evaluation environments for Model Context Protocol tool performance
Benchmarking custom MCP servers against industry-standard datasets like SWE-bench

About

Key Features

Automated generation of valid mcpbr YAML configuration templates
Validation of critical {workdir} placeholders to prevent runtime benchmarking errors
20 GitHub stars
Pre-configured templates for SWE-bench, CyberGym, and MCPToolBench++
Support for simplified model aliases and secure environment variable handling
Verification of system executables including npx, uvx, and python

Use Cases

Validating configuration files before running large-scale benchmark jobs to save time and API costs
Setting up standardized evaluation environments for Model Context Protocol tool performance
Benchmarking custom MCP servers against industry-standard datasets like SWE-bench