接続

Skill Benchmarking & Comparison

merceralex397-collabbymerceralex397-collab
•
2
•
セキュリティとテスト

Conducts rigorous head-to-head comparisons between skill variants to determine the most effective implementation based on quantitative metrics and usefulness scores.

The Skill Benchmarking tool provides a structured framework for evaluating multiple versions of a Claude Code skill to ensure measurable improvement. By analyzing pass rates, routing accuracy, and LLM-judged usefulness scores across identical test cases, it identifies whether prompt refinements have genuinely improved quality or if specific variants should be deprecated. This utility is essential for developers performing A/B testing on skill logic, allowing for data-driven decisions on whether to keep, merge, or archive specific skill implementations.

主な機能

01Standardized benchmarking reports with clear keep/discard recommendations
02Head-to-head performance comparison for A/B testing
03Routing accuracy and pass rate metric collection
042 GitHub stars
05Statistical significance analysis to distinguish real improvements from noise
06Automated usefulness scoring using multi-dimensional LLM judging

ユースケース

01Measuring the impact of a prompt refinement or skill update
02Deciding between two competing skill variants for the same specialized task
03Auditing a skill library to prune underperforming or redundant versions