How do I add a new data quality check using this skill?

The skill provides a 5-step process: define the check function in data/dq.py, return the standard status dictionary, register it in DQRunner, update the skill documentation, and deploy using the specified tags.

Which exchanges are supported by these data quality checks?

The checks are tailored for market data feeds from Kalshi, Kraken, Kraken Futures, and Polymarket.

What does the parquet_vs_jsonl check measure?

It is a zero-tolerance accountability check that verifies every JSONL line is accounted for in the resulting Parquet files, handling both exact matches and 1:N fanout types.

What is the primary purpose of the ssmd-dq-checks skill?

It serves as a reference catalog for 13 specific data quality checks used to monitor and validate market data pipelines, helping users troubleshoot failures and understand data integrity rules.

SSMD Data Quality Inspector

Name: SSMD Data Quality Inspector
Author: aaronwald

byaaronwald

•

Analytics & Monitoring

Provides a comprehensive reference for 13 data quality checks to monitor, validate, and troubleshoot SSMD market data pipelines.

This skill serves as a specialized knowledge base for the SSMD (Statistical Sideloading Market Data) pipeline's data quality suite. It provides detailed documentation for 13 distinct checks, covering critical metrics such as record completeness, schema validation, NATS sequence continuity, and connection uptime. It is an essential resource for data engineers investigating pipeline failures, understanding complex fanout rules for exchange data (Kalshi, Kraken, Polymarket), or implementing new validation logic within the data/dq.py framework to ensure high-integrity Parquet datasets.

Key Features

01Detailed reference for 13 core Data Quality (DQ) checks including schema and null rates

02Documentation of 1:N fanout rules for complex market data message types

031 GitHub stars

04Standardized procedures for developing and deploying new DQ check functions

05Troubleshooting guides for common failure modes and exchange-specific issues

06Comprehensive mapping of pass, warn, and fail criteria for automated monitoring

Use Cases

01Investigating the root cause of 'fail' statuses in production data quality reports

02Onboarding developers to the SSMD data layout and validation infrastructure

03Auditing data accountability between raw JSONL sources and processed Parquet files

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add aaronwald/dlawskillz ssmd-dq-checks

For use in Claude.ai and ChatGPT

Download Skill