What types of graders are supported?

Eval Harness supports deterministic code-based graders (like grep or test runners), model-based graders using LLMs, and manual human-review markers.

What is Evaluation-Driven Development (EDD)?

EDD is a methodology where you define the expected behavior and success criteria for an AI agent before starting the actual implementation, similar to Test-Driven Development (TDD).

Can I use this for regression testing?

Yes, the skill includes specific workflows for regression evaluations to ensure that new changes or model updates don't break existing functionality.

Where are evaluation results stored?

Results, logs, and baselines are stored locally within the .claude/evals/ directory of your project for easy version control.

How does the pass@k metric work?

It measures the probability that the AI succeeds at least once within 'k' attempts, providing a more realistic view of reliability for stochastic AI outputs.

Eval Harness

Name: Eval Harness
Author: Infopibe

byInfopibe

•

Security & Testing

Implements Evaluation-Driven Development (EDD) to formalize success criteria and track reliability metrics for AI-assisted coding tasks.

Eval Harness is a formal evaluation framework designed to enhance Claude Code workflows by applying Evaluation-Driven Development (EDD) principles. It enables developers to define rigorous success criteria before writing code, measure agent reliability using pass@k metrics, and maintain regression test suites to prevent performance drift. By standardizing how capabilities are tested and reported, this skill transforms experimental AI interactions into a predictable, metric-driven development process suitable for production-grade software engineering.

Key Features

01Automated code-based, model-based, and human graders

02Defined Evaluation-Driven Development (EDD) workflows

03Standardized evaluation reporting and artifact storage

041 GitHub stars

05Pass@k and Pass^k reliability metric tracking

06Regression testing suites for baseline performance comparison

Use Cases

01Benchmarking Claude's performance across different model versions

02Validating security-critical paths through human-in-the-loop reviews

03Setting success criteria before initiating complex feature development

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add infopibe/everything-claude-code eval-harness

For use in Claude.ai and ChatGPT

Key Features

01Automated code-based, model-based, and human graders

02Defined Evaluation-Driven Development (EDD) workflows

03Standardized evaluation reporting and artifact storage

041 GitHub stars

05Pass@k and Pass^k reliability metric tracking

06Regression testing suites for baseline performance comparison

Use Cases

01Benchmarking Claude's performance across different model versions

02Validating security-critical paths through human-in-the-loop reviews

03Setting success criteria before initiating complex feature development

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add infopibe/everything-claude-code eval-harness

For use in Claude.ai and ChatGPT