How does the skill identify a flaky test?

It analyzes CI history to find tests with inconsistent outcomes on the same commit and scans source code for patterns known to cause non-determinism, such as race conditions, hardcoded timeouts, and shared state.

What is the 'Quarantine' process?

It is a workflow where detected flaky tests are automatically marked to be skipped in CI while being tracked in a registry, preventing them from blocking PRs until a permanent fix is verified.

What test frameworks does this skill support?

The skill is language-agnostic but features specialized patterns and workflows for JavaScript/TypeScript (Vitest/Jest), Python, and GitHub Actions environments.

Can this skill fix the flaky tests automatically?

This skill focuses on detection and isolation, but it is designed to work with the 'flaky-fix' skill to suggest repairs and uses re-run analysis to verify if a fix resolved the non-determinism.

Flaky Test Detection

Name: Flaky Test Detection
Author: jmagly

byjmagly

•

Seguridad y Pruebas

Identifies non-deterministic test failures by analyzing CI history and code patterns to improve software reliability.

The Flaky Detect skill helps developers eliminate intermittent test failures that undermine CI stability and developer productivity. Built on research foundations from Google and recent ML advancements, it analyzes historical CI logs to identify inconsistent results, scans test code for risky patterns like unawaited promises or hardcoded delays, and automates the quarantine process for unreliable tests. This skill provides a structured workflow to detect, isolate, and eventually fix tests that pass or fail without code changes, saving teams hours of manual debugging and build fatigue.

Características Principales

01Static code analysis for timing, async, and environmental flakiness patterns

02Integration with recovery tools to suggest and verify automated fixes

03Automated CI history analysis for pass/fail inconsistency detection

04Comprehensive reporting with pass rates and root cause classification

0567 GitHub stars

06Structured 'Quarantine' workflow to isolate unreliable tests from the pipeline

Casos de Uso

01Debugging mysterious intermittent failures that occur randomly across PRs

02Auditing a legacy test suite to improve CI/CD stability and trust

03Implementing a formal test quarantine process before a major production release

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jmagly/ai-writing-guide flaky-detect

For use in Claude.ai and ChatGPT

Download Skill