What is Eval-Driven Development (EDD)?

EDD is a methodology where evaluations (evals) are treated as unit tests for AI development, defining success criteria before code implementation to ensure accuracy and reliability.

Where are the evaluation definitions stored?

Evals are stored directly in your project under the .claude/evals/ directory, allowing them to be version-controlled alongside your source code.

Can I use manual human review with this skill?

Yes, the framework includes a specific 'Human Review Required' tag for manual auditing of high-risk or subjective changes that cannot be fully automated.

What types of graders are supported?

It supports code-based graders (shell scripts/grep), model-based graders (using AI to evaluate outputs), and manual human grading for complex tasks.

How does the pass@k metric work?

It measures the probability that at least one successful result is achieved within 'k' attempts, helping developers gauge the reliability of AI-generated solutions.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: unju-ai

byunju-ai

0•

Security & Testing

Implements Eval-Driven Development (EDD) to rigorously test and validate AI-generated code through formal capability and regression frameworks.

The Eval Harness skill provides a robust framework for Claude Code sessions based on Eval-Driven Development (EDD) principles. It enables developers to treat evaluations as unit tests for AI, defining success criteria before implementation, tracking regressions across changes, and measuring reliability using metrics like pass@k. With support for code-based, model-based, and human-in-the-loop graders, it ensures that AI-generated features meet high quality and security standards while preventing existing functionality from breaking during the development process.

Key Features

01Automatic regression testing to ensure stability across SHA checkpoints

02Formal Eval-Driven Development (EDD) workflow integration

03Advanced reliability metrics like pass@k and pass^k for consistency measurement

040 GitHub stars

05Multi-modal grading including deterministic code checks and AI model evaluation

06Structured reporting with detailed pass/fail logs and status summaries

Use Cases

01Measuring the reliability of Claude's output across multiple attempts for critical system paths

02Defining expected behavior and success criteria before starting complex feature implementations

03Automating quality assurance for AI-generated code to prevent functional regressions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add unju-ai/ecc eval-harness

For use in Claude.ai and ChatGPT