What should I do if my eval shows 'pending: true' with empty output?

This usually indicates a malformed API response. Use the skill to restart the evaluation; the S3 buffer will allow Inspect to resume from where it left off.

How do I check if an eval is genuinely stuck vs. just retrying?

Use the 'hawk status' command via this skill to check JSON reports. If success counts are increasing despite some failures, the evaluation is progressing; consistent failure patterns require intervention.

Does this skill help with Memory Exhaustion (OOM) errors?

Yes, it identifies OOMKilled status in pod logs and provides guidance on how to increase memory limits for the evaluation runner.

How can I test if the API proxy is the cause of the hang?

The skill provides specific curl commands to test the middleman auth proxy directly, allowing you to isolate whether issues reside with the provider or the internal infrastructure.

Inspect AI Evaluation Debugger

Name: Inspect AI Evaluation Debugger
Author: METR

byMETR

•

分析と監視

Diagnoses and resolves stalled, hanging, or failing AI evaluations within the Hawk and Inspect AI framework.

The debug-stuck-eval skill provides a specialized diagnostic workflow for developers and researchers using the UK AISI's Inspect platform. It automates the verification of authentication, retrieval of detailed status reports, and analysis of logs for specific error patterns like rate limits, token overflows, or OOM errors. This skill is particularly useful when evaluation sets appear frozen, samples aren't completing, or the system returns persistent 500 errors, allowing users to perform direct API testing and implement recovery strategies to resume progress without losing work.

主な機能

01Authentication verification and access token management

0224 GitHub stars

03Direct API proxy testing via pre-configured curl templates

04Recovery workflows for stuck evaluation sets using S3 buffers

05Log analysis for specific Inspect AI and OpenAI SDK error patterns

06Automated evaluation status and pod health monitoring

ユースケース

01Identifying the root cause of frequent 500 errors or retry loops during model testing

02Testing API connectivity and latency through middleman auth proxies

03Troubleshooting an evaluation that has stopped progressing or is hanging indefinitely

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add metr/inspect-action debug-stuck-eval

For use in Claude.ai and ChatGPT

主な機能

01Authentication verification and access token management

0224 GitHub stars

03Direct API proxy testing via pre-configured curl templates

04Recovery workflows for stuck evaluation sets using S3 buffers

05Log analysis for specific Inspect AI and OpenAI SDK error patterns

06Automated evaluation status and pod health monitoring

ユースケース

01Identifying the root cause of frequent 500 errors or retry loops during model testing

02Testing API connectivity and latency through middleman auth proxies

03Troubleshooting an evaluation that has stopped progressing or is hanging indefinitely

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add metr/inspect-action debug-stuck-eval

For use in Claude.ai and ChatGPT