How does the pass@k metric help improve AI performance?

It measures the probability of success across multiple attempts. Tracking pass@1 versus pass@3 helps developers understand if a failure is due to a fundamental capability gap or simple stochastic variance.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define the expected behavior and success criteria of an AI task before the AI begins implementation, ensuring clear goals and measurable outcomes.

Can this skill be used for regression testing?

Yes, it includes specialized regression eval types designed to compare current agent behavior against established baselines to prevent functionality drift.

Where are evaluation results and logs stored?

Evals are treated as first-class project artifacts and are stored within your repository in the .claude/evals/ directory for version control and history tracking.

What types of graders are supported?

The harness supports deterministic Code-Based Graders (bash/tests), Model-Based Graders (LLM-as-judge), Rule-Based Graders (regex), and Human-in-the-loop graders for manual review.

AI Eval Harness

Name: AI Eval Harness
Author: oabdelmaksoud

byoabdelmaksoud

0•

セキュリティとテスト

Implements eval-driven development (EDD) to measure, benchmark, and improve AI agent reliability within Claude Code sessions.

AI Eval Harness provides a structured framework for implementing Eval-Driven Development (EDD) within Claude Code. It allows developers to define success criteria before implementation, track performance using pass@k metrics, and run regression suites to ensure agent stability across model versions or prompt changes. By supporting deterministic code-based graders, model-based evaluations, and manual human reviews, it transforms AI-assisted coding from a probabilistic process into a measurable, professional engineering discipline that prioritizes reliability and regression prevention.

主な機能

010 GitHub stars

02Automated regression test suite generation and history logging

03Multi-modal grading system (Code-based, LLM-as-judge, and Human review)

04Eval-Driven Development (EDD) workflow integration

05Standardized evaluation artifact storage within project directories

06Comprehensive reliability metrics including pass@k and pass^k tracking

ユースケース

01Creating automated regression suites to ensure new prompts or code changes don't break existing functionality

02Defining objective pass/fail criteria for complex feature implementations before coding begins

03Benchmarking task completion rates and reliability across different Claude model versions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add oabdelmaksoud/superclaw eval-harness

For use in Claude.ai and ChatGPT

主な機能

010 GitHub stars

02Automated regression test suite generation and history logging

03Multi-modal grading system (Code-based, LLM-as-judge, and Human review)

04Eval-Driven Development (EDD) workflow integration

05Standardized evaluation artifact storage within project directories

06Comprehensive reliability metrics including pass@k and pass^k tracking

ユースケース

01Creating automated regression suites to ensure new prompts or code changes don't break existing functionality

02Defining objective pass/fail criteria for complex feature implementations before coding begins

03Benchmarking task completion rates and reliability across different Claude model versions