What is the difference between planted and known findings?

Planted findings are specific issues that a skill must detect to pass, while known findings are legitimate issues that aren't mandatory but shouldn't be counted as false positives.

Can I run tests for a single fixture?

Yes, if you provide a specific fixture name, the skill enters single-fixture mode, allowing for faster iteration and debugging without running the entire suite.

How does the skill handle severity mismatches?

If a skill finds a correct issue but assigns the wrong severity, it is logged as a mismatch. This counts as a 'miss' for scoring purposes but helps developers calibrate the skill's judgment.

How do I define which skill to test?

The tester uses a discovery protocol that looks for a .skill-target file in your project root or scans the _skill-configs directory for valid configurations.

Universal Skill Tester

Name: Universal Skill Tester
Author: jenyapoyarkov

byjenyapoyarkov

0•

Security & Testing

Executes automated functional test harnesses against Claude Code skills to validate accuracy and prevent regressions.

The Universal Skill Tester provides a standardized framework for evaluating the performance and reliability of Claude Code skills. By running fixtures through a skill's specific review protocol and comparing results against a two-tier system of 'planted' and 'known' findings, it offers precise metrics on detection rates and false positives. This skill is essential for developers who need to maintain high-quality AI agents, allowing them to catch logic regressions, calibrate severity levels, and identify content gaps through automated, keyword-matched reporting.

Key Features

01Two-tier scoring system distinguishing essential 'planted' issues from 'known' additional findings.

02Continuous improvement workflow that identifies skill gaps and documentation needs.

03Detailed diagnostic reporting including severity mismatch tracking and aggregate pass/fail scores.

040 GitHub stars

05Automated configuration discovery via .skill-target files and structured config directories.

06Granular keyword and location-based matching to ensure high-fidelity test validation.

Use Cases

01Performing regression testing after updating a skill's logic or reference documentation.

02Calibrating severity levels (CRIT/WARN/INFO) to ensure consistent AI feedback.

03Verifying the accuracy of a new skill against a ground-truth set of test fixtures.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jenyapoyarkov/bb-pipelines-author-skill skill-tester

For use in Claude.ai and ChatGPT

Key Features

01Two-tier scoring system distinguishing essential 'planted' issues from 'known' additional findings.

02Continuous improvement workflow that identifies skill gaps and documentation needs.

03Detailed diagnostic reporting including severity mismatch tracking and aggregate pass/fail scores.

040 GitHub stars

05Automated configuration discovery via .skill-target files and structured config directories.

06Granular keyword and location-based matching to ensure high-fidelity test validation.

Use Cases

01Performing regression testing after updating a skill's logic or reference documentation.

02Calibrating severity levels (CRIT/WARN/INFO) to ensure consistent AI feedback.

03Verifying the accuracy of a new skill against a ground-truth set of test fixtures.