How are evaluation results stored?

Evals are stored as version-controlled artifacts within your project directory, typically under .claude/evals/, including definitions and execution logs.

What are pass@k and pass^k metrics?

Pass@k measures the probability of success within 'k' attempts, while pass^k requires 'k' consecutive successes, used for verifying high-reliability critical paths.

Does this skill support automated testing?

Yes, it includes code-based evaluators that can run shell commands, grep for patterns, and execute test suites to verify code deterministically.

What is Eval-Driven Development (EDD)?

EDD is a development philosophy that treats evaluations as unit tests for AI. You define success criteria before implementation to guide the AI and measure its reliability.

Eval-Driven Development Harness

Name: Eval-Driven Development Harness
Author: Nixdorfer

byNixdorfer

•

보안 및 테스팅

Implements a formal evaluation framework for Claude Code sessions using eval-driven development (EDD) principles to ensure code reliability.

The Eval-Driven Development Harness provides a structured framework for Claude Code, enabling developers to treat evaluations as unit tests for AI-assisted coding. By defining expected behaviors before implementation and tracking reliability via pass@k metrics, it allows for quantifiable measurement of AI performance. The skill supports deterministic code-based grading, model-based evaluation for subjective outputs, and manual human review triggers, ensuring that every AI-generated change is validated against specific capability and regression benchmarks.

주요 기능

01Support for pass@k and pass^k reliability metrics

02Formalized Eval-Driven Development (EDD) workflow

03Standardized templates for capability and regression evaluations

04Automated regression testing with baseline tracking

051 GitHub stars

06Multi-modal grading including code-based and model-based evaluators

사용 사례

01Measuring the statistical reliability of Claude's output across multiple attempts

02Defining success criteria and edge cases before initiating complex AI code generation

03Verifying that new AI-implemented features do not break existing project functionality

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nixdorfer/claudecodetool eval-harness

For use in Claude.ai and ChatGPT

Download Skill