How do I handle OOMKilled status in an evaluation pod?

OOMKilled indicates memory exhaustion; you should increase the pod memory limits in your evaluation configuration and restart the task.

Will restarting a stuck evaluation lose my progress?

No, because the Hawk platform uses S3 buffers to store progress, allowing the Inspect framework to resume from the last successful sample.

What does a 'Pod UID mismatch' error signify?

This error indicates a sandbox pod was killed and restarted. Usually, no manual fix is needed as Inspect AI will automatically retry the sample.

How do I identify a stuck evaluation?

A stuck evaluation is typically indicated by messages like 'eval not progressing', 'samples not completing', or abnormally high HTTP retry counts in the logs.

How can I test if the middleman proxy is causing failures?

You can use the skill to run curl commands that compare the response from the middleman internal proxy against direct provider API calls.

Inspect AI Eval Debugger

Name: Inspect AI Eval Debugger
Author: METR

byMETR

•

分析と監視

Troubleshoots and resolves hanging or failing AI model evaluations running on the Hawk platform.

This skill provides a comprehensive toolkit for diagnosing and fixing stuck or failing AI evaluations within the METR/Inspect ecosystem. It enables Claude to navigate the Hawk CLI to verify authentication, monitor evaluation status, analyze log patterns for specific API errors, and perform direct connectivity tests. By identifying common failure modes like rate limits, memory exhaustion, or proxy issues, it helps developers recover stalled experiments and ensure reliable model performance assessment using UK AISI's Inspect framework.

主な機能

0124 GitHub stars

02Direct API connectivity testing to isolate issues between proxy layers and providers

03Automated diagnosis of stuck evaluation sets using Hawk CLI status reports

04Resource monitoring to detect OOM kills and sandbox pod synchronization issues

05Deep log analysis to identify specific error patterns and API retry loops

06Recovery workflows for restarting stalled evaluations using S3 resume buffers

ユースケース

01Investigating why specific samples or eval sets are frozen or not progressing

02Debugging authentication and connectivity issues with the Hawk infrastructure

03Resolving 500 errors and timeouts during large-scale model evaluation runs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add metr/inspect-action debug-stuck-eval

For use in Claude.ai and ChatGPT

主な機能

0124 GitHub stars

02Direct API connectivity testing to isolate issues between proxy layers and providers

03Automated diagnosis of stuck evaluation sets using Hawk CLI status reports

04Resource monitoring to detect OOM kills and sandbox pod synchronization issues

05Deep log analysis to identify specific error patterns and API retry loops

06Recovery workflows for restarting stalled evaluations using S3 resume buffers

ユースケース

01Investigating why specific samples or eval sets are frozen or not progressing

02Debugging authentication and connectivity issues with the Hawk infrastructure

03Resolving 500 errors and timeouts during large-scale model evaluation runs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add metr/inspect-action debug-stuck-eval

For use in Claude.ai and ChatGPT