Eval Harness FAQs

Question 1

How does Eval Harness measure AI reliability?

Accepted Answer

It uses pass@k metrics (at least one success in k attempts) and pass^k (all k attempts succeed) to quantify the consistency and reliability of Claude's output over multiple trials.

Question 2

Can I use this for regression testing?

Accepted Answer

Yes, the framework includes specific templates for Regression Evals designed to compare current performance against a known baseline or SHA to prevent functional regressions.

Question 3

What kind of graders does this skill support?

Accepted Answer

It supports deterministic code-based graders (like grep or npm test), model-based graders where Claude reviews open-ended output, and human-in-the-loop graders for manual verification.

Question 4

What is Eval-Driven Development (EDD)?

Accepted Answer

EDD is a methodology where you define expected AI behavior through formal evaluations before the implementation phase, similar to how Test-Driven Development (TDD) works for traditional software engineering.

Question 5

Where are the evaluation definitions and results stored?

Accepted Answer

Evals are stored directly in your project under the .claude/evals/ directory, ensuring that your evaluation criteria are versioned and shared alongside your source code.

Eval Harness

Key Features

Use Cases

Eval Harness

Key Features

Use Cases