How does the multi-agent analysis work?

The skill triggers parallel agents to independently evaluate a document's quality, difficulty, and domain relevance, then uses a consensus aggregator to provide a final inclusion decision based on weighted scores.

How is Langfuse utilized in this skill?

It uses Langfuse to trace the curation process, logging individual dimension scores (accuracy, depth, coherence) and providing an audit trail for every document added to the dataset.

Does this skill help with data deduplication?

Yes, it includes a duplicate prevention checklist that checks source URLs and runs semantic similarity analysis against existing document embeddings before adding new entries.

What is a 'golden dataset' in AI development?

A golden dataset is a hand-curated, high-quality collection of data used as the 'ground truth' to benchmark and evaluate the performance, accuracy, and reliability of AI models.

Can I use this for RAG (Retrieval-Augmented Generation) testing?

Yes, it is specifically designed to generate the test queries and document metadata required to measure the retrieval and generation quality of RAG pipelines.

Golden Dataset Curation

Name: Golden Dataset Curation
Author: yonatangross

byyonatangross

•

データサイエンスとML

Automates the curation and quality validation of high-fidelity evaluation datasets using multi-agent analysis pipelines.

概要

The Golden Dataset Curation skill streamlines the creation of high-quality benchmarks for AI evaluation by implementing a rigorous multi-agent validation workflow. It automatically classifies content types, assigns semantic difficulty levels, and generates targeted test queries to ensure dataset diversity and rigor. By utilizing a weighted consensus model and integrating directly with Langfuse for observability, this skill provides a production-ready framework for maintaining the 'ground truth' data necessary for RAG systems, LLM fine-tuning, and performance testing.

主な機能

Difficulty stratification from trivial to adversarial edge cases
Native Langfuse integration for detailed quality tracing and scoring
Automated test query generation for comprehensive document coverage
Automatic content classification across multiple technical domains
Multi-agent validation pipeline with weighted consensus logic
29 GitHub stars

ユースケース

Building a benchmark dataset for evaluating RAG system retrieval and accuracy
Automating quality gates for adding new technical documentation to AI training sets
Stratifying test data by complexity to identify model performance regressions

概要

主な機能

Difficulty stratification from trivial to adversarial edge cases
Native Langfuse integration for detailed quality tracing and scoring
Automated test query generation for comprehensive document coverage
Automatic content classification across multiple technical domains
Multi-agent validation pipeline with weighted consensus logic
29 GitHub stars

ユースケース

Building a benchmark dataset for evaluating RAG system retrieval and accuracy
Automating quality gates for adding new technical documentation to AI training sets
Stratifying test data by complexity to identify model performance regressions