What metrics does the coverage analysis provide?

It provides insights into domain distribution, document types (tutorials vs research), and ensures a balanced ratio of query difficulties from trivial to adversarial.

Can I integrate this into my CI/CD pipeline?

Yes, this skill is designed to work with pre-commit hooks and data pipelines to ensure that every document added to your repository meets your quality standards automatically.

What is a golden dataset in AI development?

A golden dataset is a high-quality, human-verified set of data used as a ground truth for evaluating AI model performance, RAG accuracy, and benchmarking results.

How does this skill detect duplicate content?

The skill uses similarity thresholds to identify content that is semantically or structurally too similar to existing entries, blocking additions with similarity scores above 0.90.

Golden Dataset Quality Validator

Name: Golden Dataset Quality Validator
Author: yonatangross

byyonatangross

•

Data Science & ML

Ensures high-quality AI evaluation datasets by performing automated schema validation, duplicate detection, and coverage analysis.

This skill provides a comprehensive suite of validation tools specifically designed for maintaining 'golden datasets' used in AI model evaluation and RAG benchmarking. It automates the process of checking document integrity against strict JSON schemas, identifies near-duplicate content using semantic similarity thresholds, and ensures balanced coverage across difficulty levels and domains. By integrating these checks into your data pipeline, you can prevent redundant entries and ensure your AI evaluation sets remain diverse, accurate, and production-ready.

Key Features

01Pre-commit hook integration for continuous data quality assurance

02Comprehensive coverage analysis across domains and difficulty levels

03Referential integrity checks between query sets and document sections

04Semantic duplicate detection with configurable similarity thresholds

05Automated JSON schema validation for documents and queries

0669 GitHub stars

Use Cases

01Validating new synthetic data entries before merging into a master evaluation set

02Auditing existing datasets to find and remove redundant or low-quality content

03Ensuring balanced difficulty distribution in benchmark queries for LLM testing

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit golden-dataset-validation

For use in Claude.ai and ChatGPT

Download Skill