What is a flaky test?

A flaky test is a test that shows both passing and failing results across multiple runs without any changes to the code or the test itself, often caused by non-deterministic factors.

Does this skill provide a final report?

Yes, it generates an analysis report including a list of flaky tests, consistency scores, failure patterns, and a hypothesis regarding the root cause.

How does this skill identify inconsistencies?

It analyzes test logs from multiple runs, using diffing and pattern matching to extract failures and calculate a consistency score for each test.

What common issues does the analyzer look for?

It specifically looks for timing issues, race conditions in setup/teardown, unseeded randomness, and resource constraints like memory or disk space differences between environments.

Can I use this for languages other than Mojo?

While configured for the Mojo training framework, the underlying logic for comparing test logs and identifying failure patterns is applicable to most terminal-based test outputs.

Test Diff Analyzer

Name: Test Diff Analyzer
Author: HomericIntelligence

byHomericIntelligence

•

보안 및 테스팅

Analyzes differences between test runs to identify flaky tests and diagnose intermittent failures in software projects.

The Test Diff Analyzer skill empowers developers to detect and resolve non-deterministic test behavior by comparing execution logs across multiple runs. It identifies inconsistent pass/fail rates, calculates consistency scores, and helps pinpoint root causes such as race conditions, resource constraints, or randomness issues. By automating the comparison of local and CI results, it streamlines the debugging of environment-specific failures and improves overall test suite reliability within the Mojo ecosystem and beyond.

주요 기능

0114 GitHub stars

02Automated log comparison between multiple test iterations

03Integration with Mojo and Pixi workflows for specialized training frameworks

04Pattern detection for timing issues, race conditions, and resource bottlenecks

05Flaky test identification with pass/fail rate consistency scoring

06Structured reporting with root cause hypotheses and remediation recommendations

사용 사례

01Auditing test suite reliability after significant code changes or refactoring

02Debugging tests that pass locally but fail intermittently in CI/CD pipelines

03Identifying non-deterministic code paths caused by unseeded randomness or race conditions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add homericintelligence/projectodyssey test-diff-analyzer

For use in Claude.ai and ChatGPT

Download Skill