Eval Harness FAQs

Question 1

What is Evaluation-Driven Development (EDD)?

Accepted Answer

EDD is a methodology where you define the expected behavior and success criteria for an AI agent before it starts writing code, providing a framework similar to Test-Driven Development (TDD) for AI sessions.

Question 2

How does Eval Harness measure success?

Accepted Answer

It uses specialized metrics like pass@k (success at least once in k attempts) and pass^k (consistent success across k attempts) to quantify the reliability of Claude's output.

Question 3

What types of evaluators does this skill support?

Accepted Answer

It supports three types: Code-based (bash scripts and grep), Model-based (using Claude to judge open-ended output), and Human evaluators (flagging items for manual review).

Question 4

Where are the evaluation results and definitions stored?

Accepted Answer

All evaluation data, including capability definitions, execution logs, and regression baselines, are stored within your project's .claude/evals/ directory.

Question 5

Is this skill compatible with existing CI/CD workflows?

Accepted Answer

Yes, because it uses standard tools like bash and grep and stores results in markdown/json, it can be integrated into broader developer workflows and automated reporting.

Eval Harness

Key Features

Use Cases

Eval Harness

Key Features

Use Cases