How do I define a new test in Eval Harness?

Tests are defined in YAML files within the 'evals/capability' or 'evals/regression' directories, specifying the input data, expected results, and success criteria.

What is Eval-driven development (EDD)?

EDD is a development methodology where code quality is quantitatively measured through evaluations (Evals) to track improvements and ensure consistency in AI-generated output.

Can this skill be used for regression testing?

Yes, Eval Harness includes specific support for regression tests to ensure that bug fixes remain effective and that new updates don't break existing functionality.

How does the pass@k metric work?

The pass@k metric calculates the probability that at least one correct solution is generated within 'k' attempts. For example, pass@1 measures first-try success, while pass@5 measures success within five trials.

Is Eval Harness compatible with automated pipelines?

Yes, it is designed for CI/CD integration, allowing you to run evaluations and generate performance reports as part of your automated build process.

Eval Harness

Name: Eval Harness
Author: tadokoro-ryusuke

bytadokoro-ryusuke

0•

Seguridad y Pruebas

Implements Eval-driven development to quantitatively measure code quality and track continuous improvements through standardized metrics.

The Eval Harness skill enables developers to implement Eval-driven development (EDD) within Claude Code. It provides a structured framework for defining capability and regression tests using YAML, tracking performance via pass@k metrics, and managing a systematic evaluation directory. This skill is essential for projects requiring high reliability, allowing for objective measurement of AI-generated code quality, verification of bug fixes, and maintenance of stability during refactoring or dependency updates.

Características Principales

010 GitHub stars

02Structured directory management for organized test suites and execution reports

03CI/CD integration patterns for automated quality gate enforcement

04Standardized YAML format for defining capability and regression tests

05Quantitative trend analysis to monitor code quality improvements over time

06Tracking of pass@k metrics to measure success probability over multiple attempts

Casos de Uso

01Verifying that new API endpoints or business logic meet specific pass@1 performance targets

02Measuring the impact of major refactoring on code reliability using objective metrics

03Preventing bug regressions by creating automated tests based on previous issue fixes

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add tadokoro-ryusuke/cc-plugins eval-harness

For use in Claude.ai and ChatGPT

Download Skill