Komninos Chatzipapas

Serving Large models - VLLM, LLAMA CPP Server, and SGLang

Komninos Chatzipapas

Updated on 24-Oct-2024 860 Views

Both Large Language Models (LLMs) and Vision-Language Models (VLMs) have exploded in popularity over the last two years. Powered by recent advancements in GPU tech, these models have been pre-trained on trillions of tokens and allow developers to easily leverage state-of-the-art AI, either by fine-tuning them or just using them outright.But how would one go about hosting these models? In this article, we'll compare 3 of the most popular solutions: vLLM, llama.cpp, and SGLang.vLLMReleased in June 2023 by researchers from UC Berkeley, vLLM is a high-performance model LLM backend based on a technique called PagedAttention. PagedAttention optimizes memory management which ...