What happens if a test fixture fails?

The Skill Tester follows a strict aggregate PASS criteria; if any single fixture fails its hit-rate threshold or exceeds its false-positive budget, the entire test run is marked as a failure.

What is the two-tier scoring system?

It separates 'planted' findings (critical issues that must be detected) from 'known additional' findings (legitimate issues that aren't the primary test target). This allows for accurate hit-rate scoring without penalizing the agent for finding extra bugs.

How does the tool stay updated with skill changes?

It includes a 'Stale Entries Rule' that automatically identifies and removes expected findings that haven't been matched in several consecutive runs, keeping your test suite clean as the target skill evolves.

How does the skill handle severity mismatches?

If a skill detects a planted issue but assigns the wrong severity (e.g., WARN instead of CRIT), it is flagged as a severity mismatch. This helps developers identify when an agent's logic is correct but its calibration needs adjustment.

Universal Skill Tester

Name: Universal Skill Tester
Author: jenyapoyarkov

byjenyapoyarkov

0•

セキュリティとテスト

Runs functional test harnesses against Claude Code skills to validate detection accuracy, catch regressions, and calibrate finding severity.

The Universal Skill Tester provides a robust, standardized framework for evaluating and refining the performance of custom Claude Code skills. By executing functional fixtures against a target skill and comparing the output to a sophisticated two-tier system of 'planted' and 'known' findings, it enables developers to quantitatively measure hit rates and false positive budgets. This tool is essential for developers building complex domain-specific agents who need to ensure their skills are performing reliably, adhering to review protocols, and maintaining high-quality outputs over time.

主な機能

01Automated config discovery protocol for seamless target skill selection and environment setup.

02Automated maintenance of test suites via the stale 'known_additional' entry cleanup rule.

03Standardized reporting format with aggregate scores, hit rates, and false positive analysis.

04Two-tier scoring system distinguishing between 'must-detect' planted issues and additional known findings.

050 GitHub stars

06Detailed severity mismatch tracking to distinguish between detection gaps and calibration errors.

ユースケース

01Calibrating severity levels (CRIT/WARN/INFO) to align with domain-specific standards.

02Benchmarking the accuracy of AI agents against a gold-standard set of fixtures and expected results.

03Validating logic changes and verifying regressions in custom Claude Code skills.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jenyapoyarkov/bb-pipelines-author-skill skill-tester

For use in Claude.ai and ChatGPT

主な機能

01Automated config discovery protocol for seamless target skill selection and environment setup.

02Automated maintenance of test suites via the stale 'known_additional' entry cleanup rule.

03Standardized reporting format with aggregate scores, hit rates, and false positive analysis.

04Two-tier scoring system distinguishing between 'must-detect' planted issues and additional known findings.

050 GitHub stars

06Detailed severity mismatch tracking to distinguish between detection gaps and calibration errors.

ユースケース

01Calibrating severity levels (CRIT/WARN/INFO) to align with domain-specific standards.

02Benchmarking the accuracy of AI agents against a gold-standard set of fixtures and expected results.

03Validating logic changes and verifying regressions in custom Claude Code skills.