Can I use this for regression testing?

Yes, the skill includes templates for regression evaluations that compare current results against a baseline SHA or checkpoint to ensure feature stability.

Where are the evaluation results stored?

Evaluation definitions, logs, and baselines are stored as first-class artifacts in the .claude/evals/ directory within your repository.

How does this skill handle non-deterministic AI outputs?

It utilizes model-based evaluators where Claude reviews its own work against structured prompts, alongside deterministic checks like grep, shell scripts, and test runners.

What are pass@k and pass^k metrics?

pass@k measures if a task succeeds at least once in 'k' attempts (e.g., pass@3), while pass^k measures if a task succeeds consistently in every single attempt for high-reliability paths.

What is Evaluation-Driven Development (EDD)?

EDD is a methodology where you define success criteria and evaluation tests for AI behavior before implementing code, acting as unit tests specifically designed for AI-driven development.

Eval Harness

Name: Eval Harness
Author: unju-ai

byunju-ai

0•

Security & Testing

Implements an Evaluation-Driven Development (EDD) framework to measure and ensure the reliability of AI-generated code.

The Eval Harness skill provides a structured framework for Claude Code sessions, enabling developers to adopt Evaluation-Driven Development (EDD) principles. It allows for the definition of capability and regression tests before implementation, uses deterministic code-based or probabilistic model-based evaluators, and tracks reliability through metrics like pass@k. This skill is essential for teams looking to formalize AI development workflows, catch regressions early, and maintain high standards for complex software features by treating evaluations as AI-native unit tests.

Key Features

01Version-controlled evaluation storage within the project directory

02Formal Evaluation-Driven Development (EDD) workflow integration

03Support for deterministic code-based and probabilistic model-based evaluators

040 GitHub stars

05Standardized evaluation reporting with PASS/FAIL status tracking

06Automated capability and regression testing with pass@k metrics

Use Cases

01Defining success criteria for a new feature before writing any implementation code

02Measuring the reliability of AI outputs over multiple iterations to ensure production-grade quality

03Verifying that complex code changes do not break existing functionality through regression baselines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add unju-ai/ecc eval-harness

For use in Claude.ai and ChatGPT