Golden Dataset Curation

关于

The Golden Dataset Curation skill streamlines the creation of high-quality benchmarks for AI evaluation by implementing a rigorous multi-agent validation workflow. It automatically classifies content types, assigns semantic difficulty levels, and generates targeted test queries to ensure dataset diversity and rigor. By utilizing a weighted consensus model and integrating directly with Langfuse for observability, this skill provides a production-ready framework for maintaining the 'ground truth' data necessary for RAG systems, LLM fine-tuning, and performance testing.

主要功能

• Difficulty stratification from trivial to adversarial edge cases
• Native Langfuse integration for detailed quality tracing and scoring
• Automated test query generation for comprehensive document coverage
• Automatic content classification across multiple technical domains
• Multi-agent validation pipeline with weighted consensus logic
• 29 GitHub stars

使用场景

• Building a benchmark dataset for evaluating RAG system retrieval and accuracy
• Automating quality gates for adding new technical documentation to AI training sets
• Stratifying test data by complexity to identify model performance regressions