How does the Eval Harness measure reliability?

It uses pass@k metrics (measuring at least one success in k attempts) and pass^k metrics (requiring consistent success across all k attempts) to quantify agent performance.

Does this replace traditional unit testing?

No, it complements traditional testing by focusing on the non-deterministic nature of AI agents, using unit tests as one of several grader types.

What is Eval-Driven Development (EDD)?

EDD is a workflow where success criteria (evals) are defined before code implementation to ensure AI outputs meet specific requirements and behaviors.

Where are the evaluation results stored?

Evaluation definitions, execution logs, and regression baselines are stored locally within your project in the .claude/evals/ directory.

Can I use custom graders with this skill?

Yes, the framework supports deterministic code-based graders (scripts), open-ended model-based graders using Claude, and manual human review flags.

Eval Harness

Name: Eval Harness
Author: Fabio29T

byFabio29T

0•

보안 및 테스팅

Standardizes AI agent evaluation using Eval-Driven Development (EDD) principles for reliable Claude Code workflows.

The Eval Harness skill provides a robust framework for implementing Eval-Driven Development (EDD) within Claude Code environments. By treating evaluations as the 'unit tests of AI development,' it enables developers to define success criteria before implementation, measure agent reliability through pass@k metrics, and maintain performance consistency with automated regression suites. This skill is essential for developers looking to benchmark agent capabilities, validate complex prompt changes, and ensure high-quality, predictable outputs across different model versions through structured grading and reporting.

주요 기능

01Standardized Eval-Driven Development (EDD) workflow

02Multi-modal grading including code-based, model-based, and human-in-the-loop

03Support for pass@k and pass^k reliability metrics

040 GitHub stars

05Standardized evaluation reporting and local storage formats

06Automated regression testing for agent and prompt versioning

사용 사례

01Creating a regression suite to ensure new prompts don't break existing functionality

02Defining pass/fail criteria for complex feature implementations before writing code

03Benchmarking agent reliability and performance across different Claude model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT

주요 기능

01Standardized Eval-Driven Development (EDD) workflow

02Multi-modal grading including code-based, model-based, and human-in-the-loop

03Support for pass@k and pass^k reliability metrics

040 GitHub stars

05Standardized evaluation reporting and local storage formats

06Automated regression testing for agent and prompt versioning

사용 사례

01Creating a regression suite to ensure new prompts don't break existing functionality

02Defining pass/fail criteria for complex feature implementations before writing code

03Benchmarking agent reliability and performance across different Claude model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT