How does the documentation evaluator calculate its score?

It uses a weighted combination of readability (based on sentence length), completeness (checking for required sections like usage and examples), and structure (presence of headings and code blocks).

Are these patterns customizable for specific domain needs?

Absolutely. The templates are designed as starting points, allowing you to modify weights, dimensions, and specific requirements to match your project's unique success criteria.

Can I use different models for the LLM-as-a-judge pattern?

Yes, the prompt evaluator uses LiteLLM and respects the JUDGE_MODEL environment variable, allowing you to use OpenAI, Anthropic, or other providers via a consistent interface.

What is the standard I/O contract for these evaluators?

Each evaluator reads a JSON object from stdin containing a 'candidate' key and writes a single line of JSON to stdout containing a 'score' (0.0 to 1.0) and optional diagnostic data.

Do I need specific tools to run the code evaluator?

The provided code evaluator pattern uses Bash and Pytest, but it can be adapted to any testing framework that can be triggered via shell script and return pass/fail counts.

Evaluator Patterns for Optimize Anything

Name: Evaluator Patterns for Optimize Anything
Author: ASRagab

byASRagab

0•

セキュリティとテスト

Generates standardized evaluator scripts for validating and scoring prompts, code, documentation, and agent instructions.

The Evaluator Patterns skill provides a comprehensive suite of runnable templates designed to automate the assessment of various AI-generated artifacts. By implementing a standardized I/O contract, these patterns allow developers to quantitatively score prompts using LLM-as-a-judge, validate code through test suite pass rates, analyze documentation via heuristic readability metrics, and simulate agent behavior across multiple scenarios. This skill is essential for anyone building optimization pipelines who needs reliable, programmatic feedback loops to refine and improve their AI outputs using the gepa framework or similar optimization tools.

主な機能

010 GitHub stars

02Standardized JSON I/O contract for seamless integration with optimization workflows

03LLM-as-a-judge rubric-based semantic scoring for prompts and instructions

04Multi-scenario simulation for agent runbooks and instruction sets

05Heuristic analysis for documentation readability, completeness, and structure

06Automated Bash and Pytest integration for measuring code pass ratios

ユースケース

01Setting up an automated feedback loop for prompt engineering and iterative refinement

02Integrating code validation into a CI/CD pipeline for AI-generated scripts and modules

03Quantifying the quality and structural integrity of technical documentation automatically

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add asragab/optimize-anything evaluator-patterns

For use in Claude.ai and ChatGPT

主な機能

010 GitHub stars

02Standardized JSON I/O contract for seamless integration with optimization workflows

03LLM-as-a-judge rubric-based semantic scoring for prompts and instructions

04Multi-scenario simulation for agent runbooks and instruction sets

05Heuristic analysis for documentation readability, completeness, and structure

06Automated Bash and Pytest integration for measuring code pass ratios

ユースケース

01Setting up an automated feedback loop for prompt engineering and iterative refinement

02Integrating code validation into a CI/CD pipeline for AI-generated scripts and modules

03Quantifying the quality and structural integrity of technical documentation automatically