How does the pass@k metric work?

The pass@k metric measures the probability that at least one success occurs within 'k' attempts, providing a statistical view of agent reliability.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define evaluation criteria and success metrics for an AI agent before starting the actual implementation, serving as unit tests for AI behavior.

Can I use this for regression testing?

Yes, the skill includes specific templates and workflows for Regression Evals to ensure that new changes or model updates don't break existing functionality.

What types of graders are supported?

It supports Code-Based Graders (deterministic checks via scripts), Model-Based Graders (Claude evaluating open-ended output), and Human Graders (manual review flags).

Eval Harness & EDD Framework

Name: Eval Harness & EDD Framework
Author: saar120

bysaar120

0•

Security & Testing

Implements eval-driven development principles to benchmark agent reliability and ensure high-quality task completion.

The Eval Harness skill provides a formal framework for implementing Eval-Driven Development (EDD) within Claude Code. It allows developers to treat evaluations as the unit tests of AI development by defining expected behaviors before implementation. With support for pass@k reliability metrics, regression testing, and multiple grader types—including deterministic code-based checks and model-based assessments—it ensures that AI-assisted workflows remain robust and predictable across different model versions and prompt iterations.

Key Features

01Automated templates for capability and regression evaluations

02Support for deterministic code-based, model-based, and human graders

030 GitHub stars

04Standardized Eval-Driven Development (EDD) workflow

05Pass@k and Pass^k metrics for reliability measurement

06Project-level eval storage and standardized reporting

Use Cases

01Setting up pass/fail benchmarks for complex AI-driven code migrations

02Creating automated regression suites to prevent prompt-induced bugs

03Measuring agent reliability and performance across different model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add saar120/skillgate-registry eval-harness

For use in Claude.ai and ChatGPT

Key Features

01Automated templates for capability and regression evaluations

02Support for deterministic code-based, model-based, and human graders

030 GitHub stars

04Standardized Eval-Driven Development (EDD) workflow

05Pass@k and Pass^k metrics for reliability measurement

06Project-level eval storage and standardized reporting

Use Cases

01Setting up pass/fail benchmarks for complex AI-driven code migrations

02Creating automated regression suites to prevent prompt-induced bugs

03Measuring agent reliability and performance across different model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add saar120/skillgate-registry eval-harness

For use in Claude.ai and ChatGPT