Scherlok
Automatically detects data quality anomalies with zero configuration, learning normal patterns to alert on deviations.
Scherlok tackles the persistent challenge of data quality issues, such as silent API changes, column NULL surges, or unexpected row count drops, which can lead to weeks of incorrect reporting before detection. Unlike traditional data quality tools that demand extensive rule definitions and YAML configurations, Scherlok employs a 'learn first, then detect' philosophy. It profiles your data to establish normal patterns for row counts, column types, NULL rates, value distributions, and freshness cadence, storing this locally. Subsequently, it continuously monitors for anomalies like volume fluctuations, freshness alerts, schema drift, or distribution shifts, automatically scoring each incident as INFO, WARNING, or CRITICAL without any manual threshold setup.