What is a Golden Dataset in this context?

A Golden Dataset is a curated set of input-output pairs (often stored in golden-dataset.yaml) that serves as the ground truth for measuring the performance and regression of your LLM workflow.

Can it help if my LLM only works on prompt examples?

Yes, it specializes in generalization testing. It guides you to create test cases that are in different categories from your prompt examples to ensure the model truly understands the instructions.

What is the assay-kit framework mentioned in this skill?

Assay-kit is a testing framework designed specifically for LLMs, focusing on identifying failure modes like hallucinations and providing structured ways to measure output quality against expected invariants.

How does this skill help with LLM hallucinations?

It provides specific failure-mode analysis and anti-hallucination scorers that strictly verify the difference between input and output, ensuring no fabricated content is generated.

LLM Evaluation Designer

Name: LLM Evaluation Designer
Author: CAPHTECH

byCAPHTECH

0•

보안 및 테스팅

Designs robust evaluation frameworks and test cases for LLM-based applications using the assay-kit framework.

The LLM Evaluation Designer skill empowers developers to build reliable AI workflows by identifying specific failure modes such as hallucinations, partial processing, and overfitting to prompt examples. It provides structured guidance for creating comprehensive golden datasets, designing multi-dimensional quality scorers, and implementing generalization tests to ensure AI models perform consistently across diverse inputs. By utilizing the assay-kit framework, this skill helps justify test case selection and ensures that LLM outputs meet production-grade quality standards through rigorous, matrix-based validation.

주요 기능

01Multi-dimensional quality scorer design (accuracy, precision, anti-hallucination)

02Golden dataset (golden-dataset.yaml) design and expansion guidance

03Comprehensive failure mode analysis for hallucinations and overfitting

040 GitHub stars

05Generalization testing patterns to prevent prompt-example bias

06Orthogonal test matrix construction using pairwise logic

사용 사례

01Designing evaluation benchmarks for new LLM agents or workflows

02Validating that a prompt is not overfitted to specific few-shot examples

03Creating automated quality scorers for monitoring production LLM outputs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add caphtech/claude-marketplace llm-eval-designer

For use in Claude.ai and ChatGPT

주요 기능

01Multi-dimensional quality scorer design (accuracy, precision, anti-hallucination)

02Golden dataset (golden-dataset.yaml) design and expansion guidance

03Comprehensive failure mode analysis for hallucinations and overfitting

040 GitHub stars

05Generalization testing patterns to prevent prompt-example bias

06Orthogonal test matrix construction using pairwise logic

사용 사례

01Designing evaluation benchmarks for new LLM agents or workflows

02Validating that a prompt is not overfitted to specific few-shot examples

03Creating automated quality scorers for monitoring production LLM outputs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add caphtech/claude-marketplace llm-eval-designer

For use in Claude.ai and ChatGPT