What is a flaky test?

A flaky test is a test that yields both passing and failing results across different runs without any changes to the source code, typically caused by timing issues, race conditions, or shared state.

Which CI platforms are supported?

The skill can parse standard CI logs from platforms like GitHub Actions and works with common testing frameworks such as Vitest, Jest, and Pytest.

What is test quarantining?

Quarantining is the process of isolating a known flaky test by temporarily skipping it in the main CI pipeline to prevent false failures while a permanent fix is being developed.

Can this skill fix the tests it identifies?

While the primary purpose is detection and diagnosis, it provides specific remediation advice based on the root cause and integrates with the flaky-fix skill for automated repairs.

How does this skill detect flakiness?

It uses a multi-layered approach: analyzing historical CI logs for inconsistent results, scanning test code for problematic patterns like un-awaited promises or hardcoded delays, and performing automated test re-runs.

Flaky Test Detector

Name: Flaky Test Detector
Author: jmagly

byjmagly

•

102

•

Security & Testing

Identifies non-deterministic test failures by analyzing CI history and source code patterns to improve build reliability.

The Flaky Test Detector skill is a specialized tool designed to diagnose and manage 'flaky' tests that pass or fail inconsistently without code changes. Built on research from Google and academic studies, it identifies high-risk patterns such as race conditions, hardcoded timeouts, and environment dependencies by scanning source code and analyzing historical CI logs. This skill helps engineering teams reduce 'CI noise,' implement effective test quarantine workflows, and maintain a high-trust automated testing environment by providing actionable remediation steps and automated re-run analysis.

Key Features

01Research-backed classification into timing, resource, and environment categories

02Standardized reporting with quarantine recommendations and suggested fixes

03102 GitHub stars

04Source code scanning for non-deterministic patterns like async/timing issues

05Automated re-run execution to isolate and confirm intermittent failures

06CI history analysis to identify inconsistent pass/fail patterns over time

Use Cases

01Auditing test suite reliability prior to major software releases

02Reducing false positives and build instability in CI/CD pipelines

03Identifying and isolating non-deterministic code during debugging sessions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jmagly/aiwg flaky-detect

For use in Claude.ai and ChatGPT