What is Eval-Driven Development (EDD)?

EDD is a methodology where evaluations and success criteria are defined before code implementation, treating AI assessments as unit tests to ensure consistent behavior.

What types of graders are supported by Eval Harness?

It supports deterministic code-based graders (scripts), model-based graders (LLM-as-judge), and manual human graders for complex or subjective outputs.

Can I use this skill to prevent regressions?

Yes, the framework includes specific regression eval templates designed to compare current output against known baselines or previous successful runs.

Where are evaluation results and logs stored?

Evaluation definitions, historical logs, and baselines are stored locally within the .claude/evals/ directory, allowing them to be versioned alongside your code.

How does the pass@k metric help in AI development?

It measures the probability that at least one success occurs within 'k' attempts, providing a realistic view of AI reliability under controlled retries.

Eval Harness

Name: Eval Harness
Author: nbossn

bynbossn

0•

Security & Testing

Implements a formal evaluation framework for Claude Code using Eval-Driven Development (EDD) principles to measure and ensure AI reliability.

The Eval Harness skill introduces a structured approach to AI-assisted development by treating evaluations as unit tests for agentic workflows. It enables developers to define success criteria before implementation, measure reliability using pass@k metrics, and maintain regression suites to ensure new changes don't degrade performance. By integrating deterministic code-based graders alongside model-based and human reviews, it provides a comprehensive toolkit for benchmarking Claude's performance across different tasks, model versions, and complex implementation patterns.

Key Features

01Regression testing with baseline tracking and versioning

020 GitHub stars

03Standardized eval artifact storage and reporting within the project

04Eval-Driven Development (EDD) workflow management

05Automated pass@k and pass^k reliability metrics

06Multi-modal grading including Code, Model-as-judge, and Human review

Use Cases

01Creating regression test suites to prevent performance drift in prompts

02Benchmarking agent performance across different Claude model versions

03Defining objective pass/fail criteria for mission-critical code generation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nbossn/claude-code-base eval-harness

For use in Claude.ai and ChatGPT

Key Features

01Regression testing with baseline tracking and versioning

020 GitHub stars

03Standardized eval artifact storage and reporting within the project

04Eval-Driven Development (EDD) workflow management

05Automated pass@k and pass^k reliability metrics

06Multi-modal grading including Code, Model-as-judge, and Human review

Use Cases

01Creating regression test suites to prevent performance drift in prompts

02Benchmarking agent performance across different Claude model versions

03Defining objective pass/fail criteria for mission-critical code generation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nbossn/claude-code-base eval-harness

For use in Claude.ai and ChatGPT