DIO

•
3
•

Routes OpenAI- and Ollama-compatible LLM backends by learning online latency, applying SLO-aware admission, and intelligently load-balancing across vLLM, SGLang, TGI, and Ollama instances without engine modifications.

DIO functions as a non-invasive control plane for multi-instance LLM serving, offering an intelligent alternative to traditional load-balancing methods like Nginx round-robin. It dynamically learns each backend's latency, including factors like KV-cache pressure and queue depth, to route requests based on minimum predicted joint cost. This gateway ensures optimal performance by maintaining session and prefix affinity to warm KV caches, while also implementing SLO-aware admission control to prevent system overload. Compatible with stock vLLM, SGLang, TGI, and Ollama engines, DIO integrates seamlessly into existing LLM infrastructure as a drop-in solution, enhancing efficiency and reliability without requiring engine patches or custom kernels.

主な機能

01Hybrid Cost Routing: Predicts and routes to the backend with minimum joint cost based on learned latency and other factors.
02Session and Prefix Affinity: Keeps multi-turn conversations on the same replica to maximize KV-cache reuse.
03OpenAI and Ollama API Compatibility: Acts as a drop-in gateway for clients using standard OpenAI or Ollama HTTP APIs.
043 GitHub stars
05SLO-aware Admission Control: Empirically gates requests to prevent overload and ensure service level objectives are met, providing `Retry-After` headers for overflow.
06Engine-Metric Fusion: Scrapes vLLM `/metrics` for real-time telemetry like KV-cache utilization and waiting-request count.

ユースケース

01Implementing intelligent routing and admission control for self-hosted LLM infrastructure.
02Load-balancing inference requests across a heterogeneous fleet of vLLM, SGLang, TGI, and Ollama engines.
03Optimizing performance and latency for multi-instance LLM serving deployments.