How does backpressure handling work in this skill?

It uses asynchronous queues and producer-consumer patterns to manage the flow between the LLM generator and the client, preventing memory leaks when the client consumes data slower than the LLM produces it.

Can I stream responses that include function or tool calls?

Yes, this skill includes specific patterns for accumulating tool call chunks while simultaneously streaming content tokens to the user, allowing for seamless agentic behavior.

Is this skill compatible with FastAPI?

Absolutely, it includes ready-to-use patterns for FastAPI and sse-starlette to create production-grade streaming endpoints that follow industry standards.

What is the benefit of using SSE for LLM responses?

SSE (Server-Sent Events) allows for a lightweight, unidirectional stream of text from the server to the client, which is significantly more efficient than polling for real-time AI output.

LLM Streaming & Real-time Patterns

Name: LLM Streaming & Real-time Patterns
Author: yonatangross

byyonatangross

•

API開発

Implements production-ready real-time token streaming, SSE endpoints, and tool-call handling for AI applications.

This skill provides standardized implementation patterns for delivering LLM responses in real-time, significantly improving user experience by reducing time-to-first-token. It covers the full lifecycle of a streaming response, from basic OpenAI async generators and FastAPI Server-Sent Events (SSE) to complex frontend consumption and tool-call accumulation. By incorporating best practices for backpressure management and error handling, it ensures your AI-driven interfaces remain responsive and robust even during high-latency model generations or complex tool-use scenarios.

主な機能

01Asynchronous token streaming patterns

0269 GitHub stars

03FastAPI Server-Sent Events (SSE) implementation

04Streaming tool-call accumulation and execution

05Frontend stream consumption and UI update logic

06Backpressure and buffer management for slow consumers

ユースケース

01Implementing real-time progress updates for long-running AI agents

02Building low-latency AI chat interfaces and dashboards

03Managing high-concurrency streaming endpoints in production

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit llm-streaming

For use in Claude.ai and ChatGPT

Download Skill