Can I use this for testing terminal-based agents?

Yes, this skill is specifically designed for agentic terminal user interface tools and similar agent-driven workflows.

How does this skill handle non-deterministic AI outputs?

It uses the [L] identifier for LLM-as-judge validation, utilizing specific, objective rubrics to assess quality where code-based checks are insufficient.

What is the difference between [C] and [G] validation types?

[C] represents deterministic code-based validation logic, while [G] indicates validation against a specific ground truth output.

What is Spec-Test-Driven Development (STDD)?

STDD is a development methodology where specifications and evaluation tests are defined upfront to drive the creation and refinement of agentic tools.

Does this skill help with reasoning trace analysis?

Absolutely. The rubric.md template is specifically designed to create criteria for evaluating an agent's reasoning process and decision-making logic.

Agent Evaluation Suite

Name: Agent Evaluation Suite
Author: craigtkhill

bycraigtkhill

Security & Testing

Standardizes the creation and maintenance of evaluation suites for AI agents using structured rubrics and validation patterns.

About

The Evaluation Skill provides a robust framework for building and managing comprehensive evaluation suites for agentic tools following a Spec-Test-Driven Development (STDD) process. It enables developers to define clear success criteria through standardized spec.md and rubric.md files, facilitating a hybrid validation approach that combines deterministic code-based checks with qualitative LLM-as-judge assessments. This skill is essential for ensuring AI agents meet specific domain requirements, maintain high-quality reasoning traces, and remain reliable throughout the development lifecycle.

Key Features

Structured requirement identification using unique REQ-EVAL naming conventions
Objective pass/fail criteria definitions for qualitative reasoning assessments
Hierarchical categorization of evaluation requirements for complex agent behaviors
0 GitHub stars
Standardized templates for evaluation specifications and reasoning rubrics
Support for Ground Truth, Code-based, and LLM-as-judge validation types

Use Cases

Refining reasoning trace rubrics to improve the accuracy of LLM-as-judge assessments
Migrating ad-hoc testing processes to a standardized Spec-Test-Driven Development framework
Building a new evaluation suite to benchmark the performance of a custom AI agent

About

Key Features

Structured requirement identification using unique REQ-EVAL naming conventions
Objective pass/fail criteria definitions for qualitative reasoning assessments
Hierarchical categorization of evaluation requirements for complex agent behaviors
0 GitHub stars
Standardized templates for evaluation specifications and reasoning rubrics
Support for Ground Truth, Code-based, and LLM-as-judge validation types

Use Cases

Refining reasoning trace rubrics to improve the accuracy of LLM-as-judge assessments
Migrating ad-hoc testing processes to a standardized Spec-Test-Driven Development framework
Building a new evaluation suite to benchmark the performance of a custom AI agent