Evaluator Patterns for Optimize Anything FAQs

Question 1

How does the documentation evaluator calculate its score?

Accepted Answer

It uses a weighted combination of readability (based on sentence length), completeness (checking for required sections like usage and examples), and structure (presence of headings and code blocks).

Question 2

Are these patterns customizable for specific domain needs?

Accepted Answer

Absolutely. The templates are designed as starting points, allowing you to modify weights, dimensions, and specific requirements to match your project's unique success criteria.

Question 3

Can I use different models for the LLM-as-a-judge pattern?

Accepted Answer

Yes, the prompt evaluator uses LiteLLM and respects the JUDGE_MODEL environment variable, allowing you to use OpenAI, Anthropic, or other providers via a consistent interface.

Question 4

What is the standard I/O contract for these evaluators?

Accepted Answer

Each evaluator reads a JSON object from stdin containing a 'candidate' key and writes a single line of JSON to stdout containing a 'score' (0.0 to 1.0) and optional diagnostic data.

Question 5

Do I need specific tools to run the code evaluator?

Accepted Answer

The provided code evaluator pattern uses Bash and Pytest, but it can be adapted to any testing framework that can be triggered via shell script and return pass/fail counts.

Evaluator Patterns for Optimize Anything

Key Features

Use Cases

Evaluator Patterns for Optimize Anything

Key Features

Use Cases