Eval Harness for Claude Code FAQs

Question 1

Where are evaluation results stored?

Accepted Answer

Evaluation definitions, execution logs, and baseline snapshots are stored locally within your project's .claude/evals/ directory, making them easy to version control.

Question 2

What types of graders are supported?

Accepted Answer

It supports deterministic code-based graders (scripts and grep), model-based graders using LLMs to judge open-ended output, and human-in-the-loop review flags for manual verification.

Question 3

What is Eval-Driven Development (EDD)?

Accepted Answer

EDD is a methodology where you define success criteria and evaluation benchmarks before implementation, ensuring AI outputs meet specific quality standards and requirements.

Question 4

Can I use this for regression testing?

Accepted Answer

Yes, the skill includes specific templates and workflows for regression evals to verify that new code changes or model updates don't break existing functionality.

Question 5

How does the pass@k metric work in this skill?

Accepted Answer

It measures the probability that at least one successful result is achieved within 'k' attempts, providing a clear, statistical picture of AI agent reliability for specific tasks.

Eval Harness for Claude Code

Key Features

Use Cases

Eval Harness for Claude Code

Key Features

Use Cases