Does it handle dependency management?

Yes, the skill automatically updates the pyproject.toml file to include the new metric in the appropriate optional dependency groups and project-wide metric lists.

What files does the Metric Creator generate?

It generates the metric implementation, Pydantic schemas, unit tests, mock data fixtures, example notebooks, and updates the project configuration files like pyproject.toml.

Can I create LLM-based metrics with this skill?

Yes, it includes specialized support for LLM-Judge metrics, providing prompts, structured output schemas, and the necessary architecture for LLM-based evaluation.

Is it compatible with custom evaluation logic?

Absolutely. It provides the architectural skeleton and data flow patterns while allowing you to implement your specific evaluation or statistical logic within the batch() method.

Fair-Forge Metric Creator

Name: Fair-Forge Metric Creator
Author: Alquimia-ai

byAlquimia-ai

0•

보안 및 테스팅

Generates standardized AI evaluation metrics for the Fair-Forge library including schemas, implementations, and comprehensive test suites.

The Fair-Forge Metric Creator is a specialized development skill designed to streamline the expansion of the Fair-Forge AI evaluation framework. It automates the boilerplate-heavy process of adding new metrics by scaffolding implementation files, Pydantic schemas, unit tests, and mock data fixtures according to established architectural patterns. Whether building simple rule-based metrics, LLM-as-a-judge evaluators, or complex statistical guardians, this tool ensures consistent structure, proper dependency management, and documentation across the entire evaluation pipeline.

주요 기능

01Generation of comprehensive unit tests and mock data fixtures

02Automated file scaffolding for Python implementations and Pydantic schemas

03Automatic dependency management updates in pyproject.toml

04Creation of example Jupyter notebooks and sample datasets for documentation

050 GitHub stars

06Support for LLM-Judge, Guardian-based, and Aggregation metric patterns

사용 사례

01Adding a new logical coherence metric to evaluate multi-turn AI conversations

02Implementing domain-specific safety checks using an LLM-as-a-judge pattern

03Extending the Fair-Forge library with custom statistical analysis for AI response factuality

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add alquimia-ai/fair-forge metric-creator

For use in Claude.ai and ChatGPT

Download Skill