What is the difference between code-based and model-based evaluators?

Code-based evaluators use deterministic tools like Grep or test runners to verify output, while model-based evaluators use Claude itself to judge open-ended or subjective results based on a scoring rubric.

What is Evaluation-Driven Development (EDD)?

EDD treats evaluations as unit tests for AI development. It involves defining success criteria before implementation, running continuous evaluations, and tracking reliability to ensure AI-generated code meets specific standards.

How do I start an evaluation with this skill?

You follow a four-step workflow: Define the evaluation (pre-coding), Implement the code, Evaluate using the /eval check command, and finally generate a Report to verify the implementation is ready for review.

How does the pass@k metric improve AI reliability?

Pass@k measures the probability that the AI succeeds at least once in 'k' attempts. This is crucial for stochastic AI models, allowing developers to set high reliability targets (e.g., pass@3 > 90%) for critical tasks.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: Fabio29T

byFabio29T

0•

セキュリティとテスト

Implements a formal Evaluation-Driven Development (EDD) framework to measure and ensure the reliability of AI-generated code changes.

The Eval Harness skill brings structured testing and metrics to Claude Code sessions by establishing a formal Evaluation-Driven Development (EDD) workflow. It allows developers to treat AI evaluations as unit tests, defining expected behaviors before implementation to ensure precision and stability. By combining deterministic code-based checks with model-based grading and reliability metrics like pass@k, it provides a rigorous method for tracking capability gains and preventing regressions in complex software projects.

主な機能

01Support for deterministic code-based and probabilistic model-based grading

020 GitHub stars

03Formal Evaluation-Driven Development (EDD) workflow implementation

04Reliability tracking using pass@k and pass^k statistical metrics

05Automated capability and regression testing frameworks

06Standardized evaluation reporting and project-level eval storage

ユースケース

01Defining success criteria before implementing new features with Claude

02Ensuring AI-driven refactoring doesn't introduce regressions in legacy code

03Measuring the consistency and reliability of AI outputs for mission-critical logic

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT

主な機能

01Support for deterministic code-based and probabilistic model-based grading

020 GitHub stars

03Formal Evaluation-Driven Development (EDD) workflow implementation

04Reliability tracking using pass@k and pass^k statistical metrics

05Automated capability and regression testing frameworks

06Standardized evaluation reporting and project-level eval storage

ユースケース

01Defining success criteria before implementing new features with Claude

02Ensuring AI-driven refactoring doesn't introduce regressions in legacy code

03Measuring the consistency and reliability of AI outputs for mission-critical logic

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add fabio29t/everything-claude eval-harness

For use in Claude.ai and ChatGPT