Monitors data quality dimensions, generates scorecards, and detects distribution drift for robust data pipelines.
The data-quality skill provides a comprehensive framework for data engineers to implement automated quality gates within their pipelines. It offers tools to measure critical dimensions such as completeness, uniqueness, and freshness while providing advanced capabilities like statistical distribution drift detection using Scipy. By generating detailed scorecards and HTML reports, it enables teams to maintain high data standards, ensure reliability in production environments, and proactively identify data staleness or logical inconsistencies before they impact downstream analytics.
주요 기능
01Comprehensive quality scorecard generation with pass/fail status
02Real-time freshness and volume monitoring for pipeline health
03Automated HTML report generation for stakeholder reporting
0418 GitHub stars
05Statistical distribution drift detection to identify data shifts
06Automated data quality dimensions including completeness, uniqueness, and validity
사용 사례
01Monitoring ML training data for feature drift and staleness
02Implementing automated quality gates in ETL/ELT data pipelines
03Generating audit-ready data quality reports for data governance
