Where are my evaluation files stored?

All evaluation definitions, execution logs, and regression baselines are stored locally within your project's .claude/evals directory for easy version control management.

What types of graders are available?

Eval Harness supports deterministic code-based graders (like grep or npm test), model-based graders for qualitative evaluation, and human-in-the-loop flags for manual review.

Can I use this for existing codebases?

Yes, the regression evaluation feature allows you to set baselines for existing functionality, ensuring that new AI-generated code does not break current features.

How does pass@k improve my workflow?

It measures how often the AI succeeds within a specific number of attempts, helping you quantify the reliability of complex tasks and identify where prompts or constraints need adjustment.

What is Eval-Driven Development (EDD)?

EDD is a development workflow where you define evaluation criteria (effectively AI unit tests) before implementation, ensuring the AI has a clear target for success and measurable performance.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: sori883

bysori883

0•

セキュリティとテスト

Implements Eval-Driven Development (EDD) principles to define, track, and validate AI-generated code quality through formal evaluation frameworks.

Eval Harness is a specialized framework for Claude Code designed to bring the rigor of software testing to AI-assisted development. By treating evaluations as 'AI unit tests,' it allows developers to define expected behaviors before coding begins, run continuous capability and regression checks during implementation, and measure reliability using metrics like pass@k. The skill supports multiple grading methodologies—from deterministic code-based checks to qualitative model-based reviews—ensuring that AI-driven changes are reliable, high-quality, and regression-free.

主な機能

01Integrated version control for evaluations via the .claude/evals directory

02Standardized templates for Capability and Regression evaluations

03Reliability tracking using pass@k and pass^k success metrics

040 GitHub stars

05Multi-modal grading including deterministic code-based and AI model-based graders

06Automated reporting and status generation for feature development

ユースケース

01Defining objective success criteria before initiating complex feature implementations

02Preventing regressions during AI-led refactoring of legacy codebases

03Measuring and improving the reliability of AI code generation across multiple attempts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add sori883/ai-settings eval-harness

For use in Claude.ai and ChatGPT