How does the skill identify a flaky test?

It analyzes CI history to find tests with inconsistent outcomes on the same commit and scans source code for patterns known to cause non-determinism, such as race conditions, hardcoded timeouts, and shared state.

What is the 'Quarantine' process?

It is a workflow where detected flaky tests are automatically marked to be skipped in CI while being tracked in a registry, preventing them from blocking PRs until a permanent fix is verified.

What test frameworks does this skill support?

The skill is language-agnostic but features specialized patterns and workflows for JavaScript/TypeScript (Vitest/Jest), Python, and GitHub Actions environments.

Can this skill fix the flaky tests automatically?

This skill focuses on detection and isolation, but it is designed to work with the 'flaky-fix' skill to suggest repairs and uses re-run analysis to verify if a fix resolved the non-determinism.

Flaky Test Detection

Name: Flaky Test Detection
Author: jmagly

byjmagly

•

Security & Testing

Identifies non-deterministic test failures by analyzing CI history and code patterns to improve software reliability.

About

The Flaky Detect skill helps developers eliminate intermittent test failures that undermine CI stability and developer productivity. Built on research foundations from Google and recent ML advancements, it analyzes historical CI logs to identify inconsistent results, scans test code for risky patterns like unawaited promises or hardcoded delays, and automates the quarantine process for unreliable tests. This skill provides a structured workflow to detect, isolate, and eventually fix tests that pass or fail without code changes, saving teams hours of manual debugging and build fatigue.

Key Features

Static code analysis for timing, async, and environmental flakiness patterns
Integration with recovery tools to suggest and verify automated fixes
Automated CI history analysis for pass/fail inconsistency detection
Comprehensive reporting with pass rates and root cause classification
67 GitHub stars
Structured 'Quarantine' workflow to isolate unreliable tests from the pipeline

Use Cases

Debugging mysterious intermittent failures that occur randomly across PRs
Auditing a legacy test suite to improve CI/CD stability and trust
Implementing a formal test quarantine process before a major production release

About

Key Features

Static code analysis for timing, async, and environmental flakiness patterns
Integration with recovery tools to suggest and verify automated fixes
Automated CI history analysis for pass/fail inconsistency detection
Comprehensive reporting with pass rates and root cause classification
67 GitHub stars
Structured 'Quarantine' workflow to isolate unreliable tests from the pipeline

Use Cases

Debugging mysterious intermittent failures that occur randomly across PRs
Auditing a legacy test suite to improve CI/CD stability and trust
Implementing a formal test quarantine process before a major production release