Eval Harness for Claude Code FAQs

Question 1

Can I use custom graders?

Accepted Answer

Yes, the skill supports deterministic code-based graders (like shell scripts), model-based graders using Claude's reasoning capabilities, and manual human review flags for high-risk changes.

Question 2

What are pass@k metrics?

Accepted Answer

Pass@k measures the probability that at least one successful result is generated within 'k' attempts, helping developers understand the statistical reliability of an AI's output for a specific task.

Question 3

How does this skill handle regression testing?

Accepted Answer

It allows you to define baseline tests that run against new changes, ensuring that added features don't break existing functionality, and reports a pass/fail status relative to previous checkpoints.

Question 4

What is Eval-Driven Development (EDD)?

Accepted Answer

EDD is a methodology where you define the expected behavior and success criteria for an AI task before writing the code, providing a clear benchmark for success similar to Test-Driven Development (TDD).

Question 5

Where are the evaluation files stored?

Accepted Answer

All evaluations are stored directly in your project under the .claude/evals/ directory, allowing them to be versioned alongside your source code as first-class artifacts.

Eval Harness for Claude Code

주요 기능

사용 사례

Eval Harness for Claude Code

주요 기능

사용 사례