Scherlok

•
14
•

Automatically detects data quality anomalies with zero configuration, learning normal patterns to alert on deviations.

Scherlok tackles the persistent challenge of data quality issues, such as silent API changes, column NULL surges, or unexpected row count drops, which can lead to weeks of incorrect reporting before detection. Unlike traditional data quality tools that demand extensive rule definitions and YAML configurations, Scherlok employs a 'learn first, then detect' philosophy. It profiles your data to establish normal patterns for row counts, column types, NULL rates, value distributions, and freshness cadence, storing this locally. Subsequently, it continuously monitors for anomalies like volume fluctuations, freshness alerts, schema drift, or distribution shifts, automatically scoring each incident as INFO, WARNING, or CRITICAL without any manual threshold setup.

主な機能

01Profiles data to detect volume, freshness, schema, NULL, distribution, and cardinality anomalies
02Flexible alerting mechanisms via Slack, Discord, Teams, email, and generic webhooks
03Zero-config, automatic anomaly detection (no rules or YAML required)
04Generates self-contained HTML dashboards for detailed reports and history
05Seamless integration with dbt for monitoring materialized models
066 GitHub stars

ユースケース

01Proactive monitoring of critical data tables and dbt models for unexpected changes
02Integrating automated data quality gates into CI/CD pipelines
03Enabling AI agents to perform automated data quality assessments via MCP