How does the pass@k metric work?

The pass@k metric measures the probability that the AI succeeds at least once within 'k' attempts, which is a standard industry method for evaluating the reliability of stochastic models.

What is Evaluation-Driven Development (EDD)?

EDD is a workflow where success criteria and evaluation frameworks are defined for the AI agent before the implementation phase begins, ensuring clear goals and measurable outcomes.

What types of graders does Eval Harness support?

It supports three types: Code-based (deterministic checks like unit tests), Model-based (using Claude to judge open-ended output), and Human (manual review flags for high-risk changes).

Where are the evaluation results stored?

Evaluations are stored locally within your project in the .claude/evals directory, including definitions, run history logs, and regression baselines.

Eval Harness

Name: Eval Harness
Author: saar120

bysaar120

0•

安全与测试

Implements Evaluation-Driven Development (EDD) to formalize and measure Claude Code's task performance and reliability.

Eval Harness provides a rigorous framework for Claude Code sessions by introducing Evaluation-Driven Development (EDD) principles to the AI coding workflow. It enables developers to define success criteria before implementation, track performance using pass@k metrics, and run both capability and regression tests to ensure reliability across model updates. By combining code-based deterministic checks, model-based grading, and human review, it transforms AI-assisted coding into a measurable, test-driven engineering discipline that prevents functional regressions.

主要功能

01Evaluation-Driven Development (EDD) workflow integration

02Reliability tracking with pass@k and pass^k metrics

03Standardized evaluation reporting and local storage

04Automated regression testing and baseline comparison

05Multi-modal grading (Code-based, Model-based, and Human)

060 GitHub stars

使用场景

01Defining clear success criteria for complex AI-driven feature implementations

02Preventing functional regressions during iterative code refactoring and updates

03Benchmarking agent reliability and stability across different Claude model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add saar120/skillgate-registry eval-harness

For use in Claude.ai and ChatGPT

主要功能

01Evaluation-Driven Development (EDD) workflow integration

02Reliability tracking with pass@k and pass^k metrics

03Standardized evaluation reporting and local storage

04Automated regression testing and baseline comparison

05Multi-modal grading (Code-based, Model-based, and Human)

060 GitHub stars

使用场景

01Defining clear success criteria for complex AI-driven feature implementations

02Preventing functional regressions during iterative code refactoring and updates

03Benchmarking agent reliability and stability across different Claude model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add saar120/skillgate-registry eval-harness

For use in Claude.ai and ChatGPT