Can I use this skill for regression testing?

Yes, the framework includes dedicated regression eval types to ensure that new changes or prompt updates don't break existing project functionality.

How does the Eval Harness measure reliability?

It uses metrics like pass@k (at least one success in k attempts) and pass^k (k consecutive successes) to quantify how consistently the AI solves a specific problem.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define evaluation criteria before writing code, using these 'evals' as AI-specific unit tests to ensure task completion and reliability.

What are model-based graders?

Model-based graders use Claude itself to evaluate open-ended or subjective outputs that cannot be easily verified with deterministic code or simple grep patterns.

Eval Harness

Name: Eval Harness
Author: Fabio29T

byFabio29T

0•

安全与测试

Implements eval-driven development principles to measure, benchmark, and ensure the reliability of AI-assisted coding sessions.

The Eval Harness skill introduces a formal framework for Eval-Driven Development (EDD) within Claude Code. It allows developers to define success criteria before implementation, measure agent performance using pass@k metrics, and maintain rigorous regression test suites. By treating evaluations as 'unit tests for AI,' it ensures that code changes improve functionality without breaking existing features, providing a structured workflow for benchmarking different model versions and prompt strategies directly within the terminal.

主要功能

01Standardized eval reporting and file-based storage in .claude/evals/

02Automated pass@k and pass^k reliability metric tracking

030 GitHub stars

04Support for deterministic code-based and probabilistic model-based graders

05Formal Eval-Driven Development (EDD) workflow integration

06Regression testing suites to prevent agent performance decay

使用场景

01Benchmarking agent performance across different model versions or prompts

02Automating regression testing during large-scale code refactoring

03Defining rigorous pass/fail criteria for complex feature implementations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT

主要功能

01Standardized eval reporting and file-based storage in .claude/evals/

02Automated pass@k and pass^k reliability metric tracking

030 GitHub stars

04Support for deterministic code-based and probabilistic model-based graders

05Formal Eval-Driven Development (EDD) workflow integration

06Regression testing suites to prevent agent performance decay

使用场景

01Benchmarking agent performance across different model versions or prompts

02Automating regression testing during large-scale code refactoring

03Defining rigorous pass/fail criteria for complex feature implementations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT