Can I integrate this into my existing CI/CD pipeline?

Yes, the skill includes specific patterns for GitHub Actions to automate weekly backups, pre-deployment validation, and regression testing.

How does this skill handle dataset versioning?

It implements a standardized JSON backup format that is version-control friendly, ensuring data portability while providing patterns for automated embedding regeneration upon restoration.

Does it help with duplicate detection?

Yes, it features validation rules for drift and duplicate detection, using semantic similarity thresholds to prevent redundant entries from entering the dataset.

What is a golden dataset in the context of this skill?

A golden dataset is a hand-curated, high-quality set of ground-truth data used as a gold standard to benchmark and evaluate the performance of AI models, specifically for RAG and LLM applications.

What is the 9-phase curation workflow?

It is a structured process for adding new documents that includes parallel quality analysis, bias detection, silver-to-gold conversion, and difficulty stratification for test queries.

Golden Dataset Manager

Name: Golden Dataset Manager
Author: yonatangross

byyonatangross

•

116

•

데이터 과학 및 ML

Manages the complete lifecycle of high-quality evaluation datasets for AI and ML models through automated curation, validation, and versioning.

The Golden Dataset skill provides a comprehensive framework for building and maintaining the high-fidelity benchmarks required for rigorous AI model evaluation. It automates complex workflows including multi-agent annotation, diversity analysis, and 9-phase content curation to ensure data integrity. Designed for production-ready AI development, it integrates semantic drift detection, quality scoring, and CI/CD automation to help teams prevent regression in RAG systems and LLM applications while maintaining a portable, version-controlled ground truth.

주요 기능

01CI/CD patterns for automated backups and pre-deployment validation

029-phase structured curation workflow for document ingestion

03116 GitHub stars

04Automated quality scoring and semantic drift detection

05Multi-agent annotation pipelines with consensus aggregation

06Version-controlled JSON storage with embedding regeneration patterns

사용 사례

01Establishing a ground-truth benchmark for RAG system retrieval evaluation

02Automating quality control and bias detection for AI training data

03Managing dataset versioning and disaster recovery in production AI pipelines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit golden-dataset

For use in Claude.ai and ChatGPT

Download Skill