Where are evaluation results stored?

Evaluation definitions, execution logs, and baseline snapshots are stored locally within your project's .claude/evals/ directory, making them easy to version control.

What types of graders are supported?

It supports deterministic code-based graders (scripts and grep), model-based graders using LLMs to judge open-ended output, and human-in-the-loop review flags for manual verification.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define success criteria and evaluation benchmarks before implementation, ensuring AI outputs meet specific quality standards and requirements.

Can I use this for regression testing?

Yes, the skill includes specific templates and workflows for regression evals to verify that new code changes or model updates don't break existing functionality.

How does the pass@k metric work in this skill?

It measures the probability that at least one successful result is achieved within 'k' attempts, providing a clear, statistical picture of AI agent reliability for specific tasks.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: unju-ai

byunju-ai

0•

セキュリティとテスト

Implements Eval-Driven Development (EDD) to measure, benchmark, and ensure the reliability of AI-assisted coding sessions.

The Eval Harness skill brings formal evaluation principles to Claude Code by introducing Eval-Driven Development (EDD). It allows developers to define success criteria before implementation, measure agent reliability using metrics like pass@k, and maintain high-quality codebases through capability and regression testing. By integrating code-based, model-based, and human grading systems, it provides a comprehensive framework for benchmarking AI performance and catching regressions early in the development lifecycle, treating evaluations as the 'unit tests' of AI development.

主な機能

01Formal Eval-Driven Development (EDD) framework integration

02Continuous evaluation workflow with automated reporting

030 GitHub stars

04Standardized templates for capability and regression testing

05Support for code-based, model-based, and human grading

06Automated pass@k and pass^k reliability metrics tracking

ユースケース

01Benchmarking agent performance across different model versions or prompts

02Creating automated regression suites to ensure new changes don't break existing features

03Defining pass/fail criteria for complex AI-assisted tasks before coding begins

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add unju-ai/ecc eval-harness

For use in Claude.ai and ChatGPT