How does the multi-agent analysis work?

The skill triggers parallel agents to independently evaluate a document's quality, difficulty, and domain relevance, then uses a consensus aggregator to provide a final inclusion decision based on weighted scores.

How is Langfuse utilized in this skill?

It uses Langfuse to trace the curation process, logging individual dimension scores (accuracy, depth, coherence) and providing an audit trail for every document added to the dataset.

Does this skill help with data deduplication?

Yes, it includes a duplicate prevention checklist that checks source URLs and runs semantic similarity analysis against existing document embeddings before adding new entries.

What is a 'golden dataset' in AI development?

A golden dataset is a hand-curated, high-quality collection of data used as the 'ground truth' to benchmark and evaluate the performance, accuracy, and reliability of AI models.

Can I use this for RAG (Retrieval-Augmented Generation) testing?

Yes, it is specifically designed to generate the test queries and document metadata required to measure the retrieval and generation quality of RAG pipelines.

Golden Dataset Curation

Name: Golden Dataset Curation
Author: yonatangross

byyonatangross

•

Ciencia de Datos y ML

Automates the curation and quality validation of high-fidelity evaluation datasets using multi-agent analysis pipelines.

The Golden Dataset Curation skill streamlines the creation of high-quality benchmarks for AI evaluation by implementing a rigorous multi-agent validation workflow. It automatically classifies content types, assigns semantic difficulty levels, and generates targeted test queries to ensure dataset diversity and rigor. By utilizing a weighted consensus model and integrating directly with Langfuse for observability, this skill provides a production-ready framework for maintaining the 'ground truth' data necessary for RAG systems, LLM fine-tuning, and performance testing.

Características Principales

01Difficulty stratification from trivial to adversarial edge cases

02Native Langfuse integration for detailed quality tracing and scoring

03Automated test query generation for comprehensive document coverage

04Automatic content classification across multiple technical domains

05Multi-agent validation pipeline with weighted consensus logic

0629 GitHub stars

Casos de Uso

01Building a benchmark dataset for evaluating RAG system retrieval and accuracy

02Automating quality gates for adding new technical documentation to AI training sets

03Stratifying test data by complexity to identify model performance regressions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/skillforge-claude-plugin golden-dataset-curation

For use in Claude.ai and ChatGPT

Características Principales

01Difficulty stratification from trivial to adversarial edge cases

02Native Langfuse integration for detailed quality tracing and scoring

03Automated test query generation for comprehensive document coverage

04Automatic content classification across multiple technical domains

05Multi-agent validation pipeline with weighted consensus logic

0629 GitHub stars

Casos de Uso

01Building a benchmark dataset for evaluating RAG system retrieval and accuracy

02Automating quality gates for adding new technical documentation to AI training sets

03Stratifying test data by complexity to identify model performance regressions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/skillforge-claude-plugin golden-dataset-curation

For use in Claude.ai and ChatGPT