How does the Eval Harness track reliability?

It uses pass@k metrics, which measure the probability of getting at least one successful result within a specific number of attempts, helping gauge the consistency of the AI.

Where are the evaluation results stored?

Evaluation definitions, execution logs, and baseline snapshots are stored locally within your project's .claude/evals/ directory for easy version control.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define expected AI behavior and success criteria before starting implementation, serving as unit tests for AI-driven code generation.

Can I use my own test scripts with this skill?

Yes, the skill supports code-based graders that can execute bash commands, grep for patterns, or run existing test suites like Jest or Pytest.

Does this skill replace standard unit testing?

No, it complements unit testing by adding a layer of evaluation specifically designed to handle the probabilistic nature of AI-generated code and complex functional requirements.

Eval Harness

Name: Eval Harness
Author: Nixdorfer

byNixdorfer

•

Seguridad y Pruebas

Implements a formal evaluation framework for Claude Code to enable Eval-Driven Development (EDD) and track model reliability.

The Eval Harness skill brings the principles of Eval-Driven Development (EDD) to Claude Code, allowing developers to treat evaluations as the 'unit tests' of AI development. By defining success criteria before implementation, users can quantitatively track if Claude is gaining new capabilities or regressing on existing ones. The framework supports deterministic code-based graders, model-based assessments for open-ended tasks, and human review workflows. With built-in support for pass@k metrics and structured reporting, it provides a rigorous methodology for building and maintaining complex AI-generated software systems.

Características Principales

01Statistical reliability tracking via pass@k and pass^k metrics

021 GitHub stars

03Formal Eval-Driven Development (EDD) workflow integration

04Structured eval storage and history within the project directory

05Automated capability and regression testing frameworks

06Hybrid grading system (Code-based, Model-based, and Human review)

Casos de Uso

01Automating regression testing to ensure code changes don't break existing functionality

02Measuring and improving the reliability of complex AI tasks through iterative evaluation runs

03Defining feature success criteria before the AI begins implementation to ensure alignment

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nixdorfer/claudecodetool eval-harness

For use in Claude.ai and ChatGPT

Download Skill