How do I define a new test in Eval Harness?

Tests are defined in YAML files within the 'evals/capability' or 'evals/regression' directories, specifying the input data, expected results, and success criteria.

What is Eval-driven development (EDD)?

EDD is a development methodology where code quality is quantitatively measured through evaluations (Evals) to track improvements and ensure consistency in AI-generated output.

Can this skill be used for regression testing?

Yes, Eval Harness includes specific support for regression tests to ensure that bug fixes remain effective and that new updates don't break existing functionality.

How does the pass@k metric work?

The pass@k metric calculates the probability that at least one correct solution is generated within 'k' attempts. For example, pass@1 measures first-try success, while pass@5 measures success within five trials.

Is Eval Harness compatible with automated pipelines?

Yes, it is designed for CI/CD integration, allowing you to run evaluations and generate performance reports as part of your automated build process.

Eval Harness

Name: Eval Harness
Author: tadokoro-ryusuke

bytadokoro-ryusuke

Security & Testing

Implements Eval-driven development to quantitatively measure code quality and track continuous improvements through standardized metrics.

About

The Eval Harness skill enables developers to implement Eval-driven development (EDD) within Claude Code. It provides a structured framework for defining capability and regression tests using YAML, tracking performance via pass@k metrics, and managing a systematic evaluation directory. This skill is essential for projects requiring high reliability, allowing for objective measurement of AI-generated code quality, verification of bug fixes, and maintenance of stability during refactoring or dependency updates.

Key Features

0 GitHub stars
Structured directory management for organized test suites and execution reports
CI/CD integration patterns for automated quality gate enforcement
Standardized YAML format for defining capability and regression tests
Quantitative trend analysis to monitor code quality improvements over time
Tracking of pass@k metrics to measure success probability over multiple attempts

Use Cases

Verifying that new API endpoints or business logic meet specific pass@1 performance targets
Measuring the impact of major refactoring on code reliability using objective metrics
Preventing bug regressions by creating automated tests based on previous issue fixes

About

Key Features

0 GitHub stars
Structured directory management for organized test suites and execution reports
CI/CD integration patterns for automated quality gate enforcement
Standardized YAML format for defining capability and regression tests
Quantitative trend analysis to monitor code quality improvements over time
Tracking of pass@k metrics to measure success probability over multiple attempts

Use Cases

Verifying that new API endpoints or business logic meet specific pass@1 performance targets
Measuring the impact of major refactoring on code reliability using objective metrics
Preventing bug regressions by creating automated tests based on previous issue fixes