Can I add test cases from my own issue tracker?

Yes, you can use the --add-case flag to manually add bugs from JIRA, Sentry, or GitHub PRs, or bulk import them from documented bug patterns.

What is 'Forward Validation' in this context?

Forward validation checks if the AI found issues other than the target bug that were actually fixed in later commits, proving the AI is finding real technical debt.

How does this improve my AI reviews over time?

The skill automatically analyzes misses and partial matches to extract 'Learnings'. These are stored in a file used by other skills to update and sharpen your review rules.

How does the Backtest skill verify bug detection?

It checks out the specific commit where a bug was introduced and runs your review skills against that state. It then compares the AI findings to the known bug location and description.

Review Skill Backtester

Name: Review Skill Backtester
Author: suzuki0keiichi

bysuzuki0keiichi

0•

安全与测试

Evaluates the effectiveness of AI code review skills by replaying historical bugs and measuring detection accuracy.

The Backtest skill provides a rigorous framework for validating and improving Claude-generated code review perspectives. By replaying historical codebase states where bugs were documented, it measures 'Recall' (whether known bugs were caught) and 'Precision' (whether findings were validated by subsequent fixes). This tool allows teams to move beyond vibes-based AI evaluation by using real-world project history to refine review logic, reduce noise, and ensure that automated reviews are consistently catching critical issues.

主要功能

01Forward-validation of findings against subsequent fix commits

02Bulk import of test cases from PRs, JIRA, and Sentry logs

03Dual-metric evaluation of Recall and Precision

040 GitHub stars

05Automated historical bug replay via Git commit checkout

06Structured learning extraction to auto-update review skills

使用场景

01Validating that custom review skills catch architectural regressions

02Measuring AI review noise levels to optimize developer focus

03Iterative refinement of review guidelines based on missed bug detections

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add suzuki0keiichi/claude-plugins-suzuki0keiichi backtest

For use in Claude.ai and ChatGPT

主要功能

01Forward-validation of findings against subsequent fix commits

02Bulk import of test cases from PRs, JIRA, and Sentry logs

03Dual-metric evaluation of Recall and Precision

040 GitHub stars

05Automated historical bug replay via Git commit checkout

06Structured learning extraction to auto-update review skills

使用场景

01Validating that custom review skills catch architectural regressions

02Measuring AI review noise levels to optimize developer focus

03Iterative refinement of review guidelines based on missed bug detections

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add suzuki0keiichi/claude-plugins-suzuki0keiichi backtest

For use in Claude.ai and ChatGPT