DIO
Routes OpenAI- and Ollama-compatible LLM backends by learning online latency, applying SLO-aware admission, and intelligently load-balancing across vLLM, SGLang, TGI, and Ollama instances without engine modifications.
DIO functions as a non-invasive control plane for multi-instance LLM serving, offering an intelligent alternative to traditional load-balancing methods like Nginx round-robin. It dynamically learns each backend's latency, including factors like KV-cache pressure and queue depth, to route requests based on minimum predicted joint cost. This gateway ensures optimal performance by maintaining session and prefix affinity to warm KV caches, while also implementing SLO-aware admission control to prevent system overload. Compatible with stock vLLM, SGLang, TGI, and Ollama engines, DIO integrates seamlessly into existing LLM infrastructure as a drop-in solution, enhancing efficiency and reliability without requiring engine patches or custom kernels.