Diagnoses and resolves hanging or failing LLM evaluations within the Hawk and Inspect AI ecosystem.
The debug-stuck-eval skill provides specialized diagnostics for troubleshooting stalled or error-prone AI evaluations running on UK AISI's Inspect framework in the cloud. It empowers developers to verify authentication, analyze pod states, and interpret complex log patterns such as 500 errors, rate limits, and OOM issues. By utilizing standardized Hawk CLI commands and recovery procedures, it helps distinguish between infrastructure bottlenecks, proxy failures, and model provider errors to ensure evaluation sets reach completion.
Key Features
0124 GitHub stars
02Real-time log analysis and pattern matching for common API error codes (500, 429, 400)
03Automated diagnostic workflows for identifying 'stuck' Hawk evaluation sets
04Identification of memory exhaustion (OOMKilled) and pod UID mismatches
05Comparison testing between middleman auth proxies and direct provider connectivity
06Recovery and resumption protocols using S3-backed sample buffers
Use Cases
01Investigating why an evaluation has stopped progressing or appears 'hanging' at specific samples
02Differentiating between middleman proxy failures and upstream model provider outages
03Restarting failed evaluations without losing progress using Inspect AI's resume buffers
