What happens if a contract check fails?

The skill follows a fail-fast policy, immediately stopping the optimization process and providing deterministic remediation commands to fix the infrastructure issues.

Why is the score scale limited to 0-1?

The skill enforces a 0-1 scale for all internal decision-making and gating to ensure mathematical consistency during the optimization loop, even if other scales are used for display.

Can this skill create new evaluation datasets?

No, this is a read-only skill. It validates the state of existing infrastructure. Dataset creation and lifecycle management are handled by the langfuse-analyzer tool.

Do I need langfuse-analyzer to use this skill?

Yes, this skill is specifically designed to consume the canonical evaluation state and metadata produced by langfuse-analyzer.

What is the primary purpose of the evaluation-infrastructure skill?

It acts as a gatekeeper that performs 'pre-flight' checks to ensure all necessary evaluation data, such as datasets and baseline metrics, are correctly configured before an optimization loop starts.

AI Evaluation Infrastructure Validator

Name: AI Evaluation Infrastructure Validator
Author: mberto10

bymberto10

0•

Data Science & ML

Validates evaluation contract readiness and metadata integrity before initiating agentic optimization loops.

The Evaluation Infrastructure skill serves as a critical preflight validation layer for AI agent optimization workflows. By enforcing a read-only contract with langfuse-analyzer, it ensures that datasets, judge prompts, and baseline metrics are correctly configured before any optimization cycles begin. This 'fail-fast' mechanism prevents wasted compute and invalid optimization runs by confirming that all canonical evaluation metadata—including score scales, dimensions, and thresholds—is locally available and properly formatted in the required JSON/YAML snapshots.

Key Features

01Read-only evaluation contract validation and readiness checks

020 GitHub stars

03Seamless integration with langfuse-analyzer for canonical metadata

04Deterministic remediation guidance for missing or incomplete eval states

05Normalized scoring scale enforcement (0-1) to drive optimization gating

06Automated preflight procedure for agent optimization loops

Use Cases

01Validating standardized evaluation metrics across diverse AI agent implementations

02Ensuring evaluation datasets and baselines are ready before running optimization commands

03Troubleshooting broken metadata connections between Langfuse and local agent scripts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mberto10/mberto-compound evaluation-infrastructure

For use in Claude.ai and ChatGPT

Download Skill