How do pass@k metrics help in AI development?

The pass@k metric measures the probability of at least one successful outcome within 'k' attempts, providing a realistic benchmark for the reliability of AI-generated solutions.

Does this skill support human review?

Yes, it includes a Human Grader type for flagging high-risk changes or open-ended outputs that require manual verification by a developer.

Where are the evaluation files stored?

Evaluations are stored as first-class artifacts within your project directory under .claude/evals/, allowing you to version them alongside your source code.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define expected AI behavior and evaluation criteria before implementation, treating evals as the foundation for objective success measurement in AI coding.

Can I automate the evaluation of Claude's code changes?

Yes, the skill supports Code-Based Graders that use deterministic checks like bash scripts, grep patterns, and test runners to automatically validate results.

Eval Harness

Name: Eval Harness
Author: x-cmd

byx-cmd

•

Security & Testing

Implements a formal eval-driven development framework to measure and improve AI coding performance through automated testing and pass@k metrics.

The Eval Harness skill introduces Eval-Driven Development (EDD) to Claude Code, treating evaluations as the 'unit tests' of AI-assisted programming. It allows developers to define rigorous success criteria before writing code, run automated capability and regression tests, and track reliability using statistical metrics like pass@k. By providing a structured workflow for defining, implementing, and reporting on AI tasks, it ensures that Claude's outputs are high-quality, reproducible, and safe for production environments.

Key Features

01Standardized templates for Capability and Regression evaluations

029 GitHub stars

03Automated evaluation workflows for definition, implementation, and reporting

04Statistical reliability tracking via pass@k and pass^k metrics

05Multi-modal grading including deterministic code-based and model-based graders

06Project-level eval storage for version-controlled testing artifacts

Use Cases

01Preventing regressions in legacy codebases during AI-driven refactoring

02Measuring the consistency and reliability of AI agents across multiple attempts

03Defining feature requirements as executable evals before generating code

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add x-cmd/skill eval-harness

For use in Claude.ai and ChatGPT

Download Skill