Which LLM providers are supported for tracing?

It provides implementation patterns for major providers including Anthropic and OpenAI, as well as frameworks like LangChain.

Does this skill log my private prompt data?

No, the skill specifically enforces an anti-pattern against logging full prompts and responses to protect PII and reduce storage overhead.

Does it support streaming LLM responses?

Yes, it includes specialized considerations for tracking chunks and Time-to-First-Token (TTFT) to measure the responsiveness of streaming calls.

Can it calculate costs for different models automatically?

Yes, it includes logic for calculating USD costs based on input/output tokens for models like Claude 3 Opus, Sonnet, Haiku, and GPT-4o.

How does this help with debugging AI agents?

By tracking retry counts, error types, and specific failure reasons, it allows developers to identify exactly where and why an agentic workflow is failing.

LLM Call Tracing & Observability

Name: LLM Call Tracing & Observability
Author: nexus-labs-automation

bynexus-labs-automation

0•

분석 및 모니터링

Instruments LLM API calls to capture high-fidelity observability data including latency, token usage, and costs.

This skill provides standardized patterns for instrumenting Large Language Model (LLM) calls within Claude Code, enabling developers to gain deep insights into model performance and operational expenses. It enforces best practices for capturing essential metadata like provider-specific spans, token counts, and error types while ensuring sensitive data remains private by avoiding full prompt logging. Whether using direct SDKs or frameworks like LangChain, this skill helps developers optimize AI agents for cost-efficiency, speed, and production reliability.

주요 기능

01Comprehensive token and cost tracking for major providers like Anthropic and OpenAI

02Integrated error handling and retry tracking for robust agent behavior analysis

03High-precision latency monitoring including Time-to-First-Token (TTFT) for streaming

04Standardized span attribute schema for consistent observability across diverse models

05Privacy-first logging patterns that exclude PII by focusing on metadata over content

060 GitHub stars

사용 사례

01Monitoring and optimizing API costs across multiple LLM providers and models

02Auditing agent reliability by tracking failure rates and specific error types in production

03Debugging performance bottlenecks and high-latency model calls in AI agent workflows

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nexus-labs-automation/agent-observability llm-call-tracing

For use in Claude.ai and ChatGPT

Download Skill