What types of graders are supported?

It supports deterministic code-based graders (using bash/grep), model-based graders (using LLM judges for open-ended output), and human review flags for manual verification.

Where are evaluation results and definitions stored?

Evaluation definitions, logs, and baselines are stored locally within your project in the .mdt/evals/ directory, allowing them to be version-controlled alongside your code.

Can I use this for regression testing?

Yes, the Eval Harness includes specific regression eval types designed to ensure that model updates, prompt changes, or codebase refactors don't break existing functionality.

How does pass@k measurement work?

pass@k measures the probability that at least one successful result is generated within k attempts, providing a quantitative metric for AI reliability and consistency.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define success criteria and evaluation benchmarks before implementing code, ensuring AI outputs meet specific quality and functional standards.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: flatrick

byflatrick

0•

安全与测试

Implements a formal evaluation framework for Claude Code workflows using Eval-Driven Development (EDD) principles.

The Eval Harness skill brings engineering rigor to AI-assisted development by treating evaluations as the unit tests of the AI era. It allows developers to define clear pass/fail criteria, capability benchmarks, and regression suites before code implementation. By utilizing various grader types—including deterministic code-based checks and probabilistic model-based judges—this skill ensures that agentic workflows remain reliable and high-performing. It is particularly useful for teams looking to measure reliability via pass@k metrics and maintain quality across different model versions or prompt iterations.

主要功能

01Reliability tracking with pass@k and pass^k metrics

02Eval-Driven Development (EDD) workflow integration

03Automated regression testing for prompt and model changes

040 GitHub stars

05Standardized reporting for capability and performance benchmarking

06Deterministic code-based and probabilistic model-based grading

使用场景

01Benchmarking Claude's performance across different model versions

02Setting up quality gates for AI-assisted feature development

03Ensuring reliability in critical agentic workflows through pass^k validation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add flatrick/mdt eval-harness

For use in Claude.ai and ChatGPT