Why is the cleaning-data skill mandatory in DataPeeker workflows?

DataPeeker adheres to the principle that cleaning is always mandatory to prevent 'garbage in, garbage out' scenarios where duplicates or outliers skew analytical results.

How does the skill detect numeric outliers?

The skill uses a Median Absolute Deviation (MAD) threshold, typically flagged at 3 MAD from the median, to identify extreme values that may require capping or exclusion.

What happens to the original raw data after cleaning?

The skill follows a non-destructive pattern where transformations are typically applied to create clean versions of tables while preserving the original raw_* tables in the database.

Can it handle similar but not identical text duplicates?

Yes, it utilizes dedicated sub-agents for near-duplicate detection that perform fuzzy matching on text columns to identify potential data entry errors.

Data Cleaning & Quality Remediation

Name: Data Cleaning & Quality Remediation
Author: tilmon-engineering

bytilmon-engineering

•

数据科学与机器学习

Performs systematic data quality remediation by detecting duplicates, handling outliers, and standardizing datasets for reliable analysis.

关于

The cleaning-data skill is a cornerstone of the DataPeeker framework, designed to transform raw, messy data into high-quality datasets ready for rigorous analysis. It automates complex quality checks including exact and near-duplicate detection via fuzzy matching, MAD-based outlier identification, and referential integrity validation. By guiding users through a structured five-phase process—from quality report review to final verification—it ensures every cleaning decision is documented with a clear rationale, minimizing bias and maximizing the validity of subsequent analytical insights.

主要功能

Referential integrity and orphaned record validation
Detailed documentation generation for audit trails
Structured 5-phase data remediation workflow
2 GitHub stars
Automated duplicate detection with fuzzy matching capabilities
MAD-based (Median Absolute Deviation) outlier analysis

使用场景

Deduplicating and standardizing customer or transaction records in SQLite
Identifying and handling extreme numeric outliers before statistical reporting
Preparing raw CSV imports for exploratory data analysis and modeling

关于

主要功能

Referential integrity and orphaned record validation
Detailed documentation generation for audit trails
Structured 5-phase data remediation workflow
2 GitHub stars
Automated duplicate detection with fuzzy matching capabilities
MAD-based (Median Absolute Deviation) outlier analysis

使用场景

Deduplicating and standardizing customer or transaction records in SQLite
Identifying and handling extreme numeric outliers before statistical reporting
Preparing raw CSV imports for exploratory data analysis and modeling