WebBench Task Evaluator

•
3
•

Evaluates the success of web automation tasks by analyzing execution traces and page states against original requirements using LLM-as-judge scoring.

The WebBench Task Evaluator provides a standardized framework for grading the performance of AI agents performing browser-based tasks. By acting as an automated judge, it examines execution traces, final page states, and extracted data to deliver a structured verdict (PASS, PARTIAL, or FAIL). This skill is essential for benchmarking web agents, offering deep insights into navigation accuracy, data extraction completeness, and infrastructure blockers like CAPTCHAs or authentication walls. It ensures objective evaluation of READ, CREATE, UPDATE, and DELETE operations without requiring manual oversight.

Características Principales

01Standardized PASS/PARTIAL/FAIL scoring rubric
02Category-specific logic for CRUD and data extraction tasks
03Multi-dimensional analysis of navigation, comprehension, and accuracy
04Structured JSON output for integration with benchmarking pipelines
05Automated identification of infrastructure blockers like paywalls and CAPTCHAs
063 GitHub stars

Casos de Uso

01Automated quality assurance for web scraping and data collection workflows
02Debugging complex browser automation failures by identifying specific requirement gaps
03Benchmarking the reliability and accuracy of web-based AI agents