Troubleshoots and resolves hanging or failing AI model evaluations running on the Hawk platform.
This skill provides a comprehensive toolkit for diagnosing and fixing stuck or failing AI evaluations within the METR/Inspect ecosystem. It enables Claude to navigate the Hawk CLI to verify authentication, monitor evaluation status, analyze log patterns for specific API errors, and perform direct connectivity tests. By identifying common failure modes like rate limits, memory exhaustion, or proxy issues, it helps developers recover stalled experiments and ensure reliable model performance assessment using UK AISI's Inspect framework.
Características Principales
0124 GitHub stars
02Direct API connectivity testing to isolate issues between proxy layers and providers
03Automated diagnosis of stuck evaluation sets using Hawk CLI status reports
04Resource monitoring to detect OOM kills and sandbox pod synchronization issues
05Deep log analysis to identify specific error patterns and API retry loops
06Recovery workflows for restarting stalled evaluations using S3 resume buffers
Casos de Uso
01Investigating why specific samples or eval sets are frozen or not progressing
02Debugging authentication and connectivity issues with the Hawk infrastructure
03Resolving 500 errors and timeouts during large-scale model evaluation runs
