Where are the evaluation results stored?

Evaluations are stored as version-controlled markdown and log files within the .claude/evals/ directory of your project.

How does the pass@k metric work?

It measures the probability that at least one successful result is achieved within 'k' attempts, helping developers gauge the reliability of non-deterministic AI outputs.

What is Eval-Driven Development (EDD)?

EDD is a methodology where expected AI behavior and success criteria are defined before implementation, functioning as unit tests for AI-assisted development.

What is the difference between pass@k and pass^k?

pass@k measures if an agent can succeed at least once in k tries, while pass^k measures stability by requiring all k trials to succeed.

Can I use my own test scripts with this skill?

Yes, the framework supports code-based graders using Bash, Grep, and standard test runners like npm or pytest to verify outputs deterministically.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: oabdelmaksoud

byoabdelmaksoud

0•

Seguridad y Pruebas

Implements Eval-Driven Development (EDD) to measure and improve AI agent reliability using formal scoring and pass@k metrics.

Eval Harness is a comprehensive framework designed to bring engineering rigor to AI-assisted coding by treating evaluations as unit tests for LLMs. It allows developers to define success criteria before implementation, run automated regression suites, and track reliability trends using pass@k and pass^k metrics. By integrating deterministic code-based graders with model-based assessments, it ensures that Claude Code tasks meet high-quality standards consistently across model versions and prompt iterations, providing a structured path from prototype to production-grade AI workflows.

Características Principales

01Reliability metrics including pass@k and stability-focused pass^k

02Multi-modal grading system (Code, Model, and Human graders)

03Automated capability and regression testing suites

04Standardized evaluation reporting and versioned storage

050 GitHub stars

06Eval-Driven Development (EDD) workflow management

Casos de Uso

01Establishing success benchmarks before starting complex AI-assisted features

02Quantifying AI agent reliability for mission-critical production codebases

03Preventing regressions when upgrading Claude model versions or system prompts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add oabdelmaksoud/superclaw eval-harness

For use in Claude.ai and ChatGPT

Características Principales

01Reliability metrics including pass@k and stability-focused pass^k

02Multi-modal grading system (Code, Model, and Human graders)

03Automated capability and regression testing suites

04Standardized evaluation reporting and versioned storage

050 GitHub stars

06Eval-Driven Development (EDD) workflow management

Casos de Uso

01Establishing success benchmarks before starting complex AI-assisted features

02Quantifying AI agent reliability for mission-critical production codebases

03Preventing regressions when upgrading Claude model versions or system prompts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add oabdelmaksoud/superclaw eval-harness

For use in Claude.ai and ChatGPT