Lemonade SDK
Serve, benchmark, and deploy large language models (LLMs) on various hardware platforms.
The Lemonade SDK facilitates serving, benchmarking, and deploying large language models (LLMs) on diverse hardware like CPU, GPU, and NPU. It comprises a server interface using the OpenAI API for local LLM integration, a Python API for seamless LLM integration into Python applications, and a CLI tool for LLM experimentation. The CLI offers prompting, accuracy measurement, benchmarking (time-to-first-token and tokens per second), and memory usage profiling capabilities.
Key Features
01OpenAI API compatible server for local LLMs
02Python API for easy integration into Python applications
03CLI for LLM prompting, accuracy measurement, benchmarking, and profiling
04NPU acceleration
05Supports PyTorch, ONNX, and GGUF frameworks
0636 GitHub stars
Use Cases
01Benchmarking LLM performance on different hardware
02Deploying LLMs locally for privacy and cost efficiency
03Integrating LLMs into Python applications using a high-level API