Can I integrate my existing unit tests with this skill?

Yes, the Code-Based Grader allows you to trigger existing test runners like npm test, pytest, or custom shell scripts to validate Claude's work.

How does the Eval Harness handle non-deterministic AI outputs?

It uses pass@k metrics to measure how often the AI succeeds across multiple attempts and utilizes model-based graders to evaluate open-ended or creative code outputs.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define the expected behavior and success criteria of an AI task before implementation begins, serving as unit tests for AI development.

When should I use a model-based grader versus a code grader?

Use code graders for deterministic checks like build success or regex patterns, and use model-based graders for qualitative assessments like code structure or edge-case handling.

Where are the evaluation results stored?

Evaluations, run logs, and baselines are stored locally within your project's .claude/evals/ directory, allowing them to be versioned alongside your code.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: Aventerica89

byAventerica89

0•

Security & Testing

Implements Eval-Driven Development (EDD) to ensure the reliability and quality of AI-generated code through systematic testing and regression tracking.

The Eval Harness skill brings the rigor of software engineering to AI-assisted development by implementing Eval-Driven Development (EDD) principles. It allows developers to define clear success criteria before implementation starts, run continuous evaluations during development, and track regressions using automated checks. By utilizing code-based, model-based, and human graders, this skill provides a structured framework for measuring Claude's performance using industry-standard metrics like pass@k, ensuring that AI-generated features meet production-grade standards and maintain system stability over time.

Key Features

01Regression evaluation suite to prevent feature breakage during refactoring

02Support for deterministic code-based graders and probabilistic model-based graders

030 GitHub stars

04Reliability metrics tracking including pass@k and pass^k success rates

05Eval-Driven Development (EDD) workflow for defining success criteria before implementation

06Structured evaluation reporting and local storage in the .claude/ directory

Use Cases

01Defining acceptance criteria for complex feature requests before implementation

02Measuring and improving the reliability of AI-generated logic for critical system paths

03Performing continuous regression testing during large-scale code migrations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add aventerica89/claude-codex eval-harness

For use in Claude.ai and ChatGPT

Download Skill