What is Evaluation-Driven Development (EDD)?

EDD is a methodology that treats evaluations as 'AI unit tests,' where expected behavior is defined before implementation and checked continuously to ensure reliability.

Where does Eval Harness store my test results?

All evaluation definitions, execution logs, and regression baselines are stored locally in the .claude/evals/ directory within your project.

How do pass@k metrics work in this skill?

Pass@k measures the probability of success within 'k' attempts. For example, pass@3 means the task was completed successfully at least once within three tries, indicating practical reliability.

Can I use this for regression testing?

Yes, Eval Harness is specifically designed to run regression tests that ensure new AI-generated changes do not break existing project functionality.

What types of graders can I use?

You can use code-based graders for deterministic checks (like exit codes), model-based graders using LLM judges for open-ended output, and human graders for high-risk manual reviews.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: saar120

bysaar120

0•

Security & Testing

Implements Evaluation-Driven Development (EDD) principles to define, run, and measure AI agent performance using pass@k metrics.

Eval Harness is a robust framework designed to bring software engineering discipline to AI-assisted development. It enables developers to implement Evaluation-Driven Development (EDD) by defining success criteria before code is even written, treating evaluations as essential AI unit tests. The skill provides tools for capability testing, regression monitoring, and quantifying reliability through advanced metrics like pass@k and pass^k. By integrating deterministic code-based checkers, LLM-based judges, and human review flags, Eval Harness ensures that your Claude Code sessions produce high-quality, reliable, and regression-free results across various tasks and model versions.

Key Features

01Hybrid grading system including code-based, model-based, and human reviewers

02Standardized evaluation storage and reporting in .claude/evals/

03Implementation of Evaluation-Driven Development (EDD) workflows

040 GitHub stars

05Automated regression testing to prevent functional degradation

06Quantifiable reliability measurement using pass@k and pass^k metrics

Use Cases

01Maintaining a persistent regression suite for complex AI-driven refactoring tasks

02Establishing rigorous success criteria for a new feature before implementation

03Benchmarking agent performance and reliability across different model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add saar120/skillgate-registry eval-harness

For use in Claude.ai and ChatGPT

Key Features

01Hybrid grading system including code-based, model-based, and human reviewers

02Standardized evaluation storage and reporting in .claude/evals/

03Implementation of Evaluation-Driven Development (EDD) workflows

040 GitHub stars

05Automated regression testing to prevent functional degradation

06Quantifiable reliability measurement using pass@k and pass^k metrics

Use Cases

01Maintaining a persistent regression suite for complex AI-driven refactoring tasks

02Establishing rigorous success criteria for a new feature before implementation

03Benchmarking agent performance and reliability across different model versions