What is a 'golden dataset' in this context?

A golden dataset is a curated collection of high-quality, verified examples used as a 'ground truth' to evaluate the performance and accuracy of AI models and RAG systems.

What is the difference between 'Silver' and 'Gold' status?

Silver status is for documents with quality scores between 0.55 and 0.74, while Gold status is reserved for high-quality documents (0.75+) that have passed all validation and bias checks.

How does it handle bias detection?

The workflow includes a dedicated phase that checks for technology, vendor, recency, and complexity biases, assigning a bias score that determines if a document needs review or a disclaimer.

How does the quality scoring work?

The skill uses four parallel agents to score documents on accuracy, coherence, depth, and relevance, providing a weighted final score and detailed explanations for each metric.

Does it support version control for datasets?

Yes, the skill implements semantic versioning (Major.Minor.Patch) and tracks every addition, update, or promotion within the dataset for easy rollbacks.

Golden Dataset Curator

Name: Golden Dataset Curator
Author: yonatangross

byyonatangross

•

データサイエンスとML

Manages the multi-agent curation of high-quality training and testing datasets with automated quality scoring and bias detection.

The add-golden skill automates the rigorous process of building 'golden datasets'—authoritative collections of high-quality examples used for LLM evaluation and RAG systems. It employs a multi-agent workflow that extracts content, performs parallel quality analysis across four specialized dimensions, detects potential biases, and tracks version history. By implementing a silver-to-gold promotion workflow, it ensures that only the most accurate and diverse documents make it into your production-ready evaluation sets, providing transparency through detailed quality score explanations.

主な機能

01Automated bias detection for technology, vendor, and recency biases

02Comprehensive 9-phase curation workflow with built-in task management

0369 GitHub stars

04Multi-agent quality analysis covering Accuracy, Coherence, Depth, and Relevance

05Two-tier Silver-to-Gold promotion system for data maturity

06Semantic versioning and rollback capabilities for dataset integrity

ユースケース

01Maintaining a diverse, bias-free dataset for fine-tuning or testing AI models

02Benchmarking LLM performance against domain-specific ground truth documentation

03Curating high-quality evaluation sets for RAG (Retrieval-Augmented Generation) applications

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit add-golden

For use in Claude.ai and ChatGPT

Download Skill