Can I use this for regression testing?

Yes, the harness includes specific regression eval types and baseline comparison tools to ensure that updates to prompts or model versions don't break existing functionality.

How does the pass@k metric work in this skill?

It measures the probability that at least one successful result is achieved within 'k' attempts, helping developers quantify the reliability of non-deterministic AI outputs.

What is Eval-Driven Development (EDD)?

EDD is a methodology where success criteria for AI tasks are defined as formal evaluations before implementation, treating them as unit tests for AI behavior.

What types of graders are supported?

The skill supports deterministic code-based graders (like grep or shell scripts), model-based graders for open-ended analysis, and manual flags for human review.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: unju-ai

byunju-ai

0•

Security & Testing

Implements eval-driven development (EDD) to measure, benchmark, and ensure the reliability of AI-assisted coding tasks.

Eval Harness provides a formal framework for Claude Code sessions that treats evaluations as the 'unit tests' of AI development. By defining success criteria before implementation, it allows developers to systematically track agent performance through capability and regression evals. The skill supports deterministic code-based grading, probabilistic model-based grading, and manual human review, providing clear metrics like pass@k to quantify reliability and prevent performance regressions during the development lifecycle.

Key Features

01Standardized Eval-Driven Development (EDD) workflow integration

02Automated pass@k and pass^k reliability metric tracking

03Automated regression testing suite for prompt and model versioning

040 GitHub stars

05Structured evaluation reporting with history logging and baselines

06Multi-modal grading including code-based, model-based, and human review

Use Cases

01Defining objective success criteria for AI agents before starting feature development

02Ensuring new prompt engineering changes do not break existing agent capabilities

03Benchmarking agent reliability and performance across different LLM model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add unju-ai/ecc eval-harness

For use in Claude.ai and ChatGPT