Waza CLI Evaluator

•
5
•

Evaluates agent skills, audits token budgets, and gates CI regressions using the Microsoft Waza CLI evaluation framework.

Streamline the evaluation and quality assurance of agent skills using the Microsoft Waza CLI framework. This skill enables developers to verify skill readiness, audit token budgets, run behavioral evaluation suites with LLM graders, compare model baselines A/B, and enforce quality regression gates in CI pipelines. It provides structured workflows for testing custom skills before submission, isolating execution environments with BYOK (Bring Your Own Key) provider configurations, and maintaining rigorous evaluation coverage across your workspace without unnecessary token burn.

Key Features

01Offline validation and token budget auditing via waza check and waza tokens
02Multi-dimensional skill quality scoring with an automated LLM judge
03Behavioral evaluation suites with LLM graders and A/B model comparisons
04Automated CI regression gating and determinism verification
05Environment isolation and Bring Your Own Key (BYOK) provider support
065 GitHub stars

Use Cases

01Enforcing automated quality and regression checks inside GitHub Actions CI/CD
02Validating skill compliance and token limits prior to workspace submission
03Benchmarking behavioral differences and performance across distinct LLM models