Course: Course 3 — LLM Fine-Tuning Masterclass Module: FTDD-09 — llama.cpp + vLLM Duration: 45 minutes Level: Senior Engineer and above Prerequisites: FT19 (Quantization), FT20 (Serving)
After completing this module, you will be able to:
Stop treating llama.cpp and vLLM as rivals. They are complementary tools for different jobs. The deployment context decides which you reach for.
llama.cpp is a C/C++ inference engine, distributed as (effectively) a single binary with no Python runtime dependency. Its defining property is hardware breadth: the same binary runs on CPU, Apple Metal, NVIDIA CUDA, AMD ROCm, and Vulkan backends. If a machine exists, llama.cpp can probably run a model on it. You compile once per platform (or download a prebuilt release) and you have a server.
This breadth is why llama.cpp is the tool for the long tail of deployment targets: the developer laptop, the edge box, the air-gapped server with whatever hardware happens to be in it, the M-series Mac doing local inference. It is also the engine underneath Ollama and LM Studio — most "local LLM" tools are llama.cpp with a wrapper. When you serve with llama.cpp's built-in server, you get an OpenAI-compatible HTTP API on whatever port you choose, on whatever hardware you have.
vLLM is a Python-based inference engine built for one thing: maximum throughput on GPUs in a serving context. Where llama.cpp optimizes for breadth and portability, vLLM optimizes for requests-per-second on NVIDIA (and increasingly AMD) GPUs in a datacenter. Its defining properties are PagedAttention (which we cover in 9.3) and continuous batching, together with multi-GPU tensor parallelism for models too large for one GPU.
vLLM is what you reach for when you have real traffic — a production endpoint serving many concurrent users — and GPUs to serve it on. It exposes an OpenAI-compatible API out of the box, which means anything written against the OpenAI SDK works against vLLM unchanged. For most cloud-deployed open-model services, vLLM is the default.
The mistake is treating them as competing options and picking a favorite. They are not. A team that ships a local desktop assistant uses llama.cpp (the user's laptop may not have a GPU; the binary must be self-contained). A team that ships a cloud API with hundreds of concurrent users uses vLLM (throughput is the constraint, and there are GPUs). Many teams use both: llama.cpp for the local/edge component and vLLM for the cloud service. The deployment context — what hardware, what network, what traffic profile — is the entire decision.
The quant format you choose is downstream of the serving engine you choose. This is non-negotiable.
GGUF (GPT-Generated Unified Format) is llama.cpp's native container. It bundles the model weights, tokenizer, and metadata in a single file. llama.cpp's quantization produces k-quants — named like Q4_K_M, Q5_K_M, Q8_0 — which are llama.cpp-specific precision schemes designed to preserve quality at low bit-widths across the heterogeneous hardware llama.cpp targets. When you download a model from HuggingFace to run in Ollama or llama.cpp, you download a GGUF.
The k-quants matter because they are the reason llama.cpp can run a usable 7B model on a consumer laptop. Q4_K_M is the widely-recommended default — a good quality/size trade-off for most models. The K-quants use block-wise scaling that adapts precision to the importance of different weight groups, which is why they outperform naive round-to-nearest quantization at the same bit width.
vLLM serves GPU-native quantization formats: AWQ (Activation-aware Weight Quantization), GPTQ (a post-training quantization method), and increasingly FP8 (the native 8-bit float format on newer NVIDIA GPUs like Hopper and Blackwell). These formats are designed for GPU execution kernels — they are optimized for the memory bandwidth and compute patterns of CUDA/ROCm, not for CPU portability.
The critical compatibility rule: you cannot serve a GGUF in vLLM, and you cannot serve an AWQ or GPTQ in llama.cpp. The formats are tied to their engines' kernels. This means the quant decision is not independent of the serving decision. You choose the server first (based on deployment context, per 9.1), and the quant format follows.
This creates two distinct export paths after fine-tuning (Layer 4 of the Steering Stack). If you are deploying to llama.cpp, you export to GGUF (using llama.cpp's convert and quantize tools, or the llama-cpp-python converter). If you are deploying to vLLM, you export to AWQ or GPTQ (using AutoAWQ or AutoGPTQ), or you serve in FP16/BF16 if you have the VRAM and want zero quantization loss, or FP8 on supported hardware. A team that ships both local and cloud will maintain two export artifacts from the same trained weights: a GGUF for the edge and an AWQ (or FP16) for the cloud.
PagedAttention and continuous batching are the two innovations that make vLLM the production GPU default.
During generation, a transformer must store the keys and values for every token it has processed so far — the KV cache. In a naive serving implementation, this cache is pre-allocated as a contiguous block per request, sized for the maximum sequence length. The problem: real requests have variable lengths, and most do not use their full allocation. The result is severe memory fragmentation and waste — a large fraction of GPU memory sits allocated but unused, capping how many concurrent requests you can serve.
PagedAttention, introduced in the vLLM paper (Kwon et al., arXiv:2309.06180, SOSP 2023), manages the KV cache like an operating system manages virtual memory. The cache is broken into fixed-size blocks (pages), and each request's cache is a mapping of logical-to-physical blocks, allocated on demand. This eliminates the fragmentation: memory is only used for tokens actually generated. The practical effect is a dramatic increase in how many concurrent requests fit on a GPU, which directly increases throughput.
Traditional batching (static batching) waits to fill a batch, processes it, then starts the next. If one request in the batch is much longer than the others, the short requests finish early and their slots sit idle until the long one finishes. Throughput suffers.
Continuous batching (sometimes called iteration-level or dynamic batching) admits new requests into the batch as soon as a slot frees — at the token/iteration level, not the batch level. As soon as one request finishes generation, a waiting request takes its place immediately. Combined with PagedAttention's efficient memory use, this means the GPU stays saturated: there is always work to do, and there is always memory for new requests. This is the second major reason vLLM achieves high throughput under concurrent load.
For models too large for a single GPU (e.g., a 70B model in FP16 needs ~140GB, beyond one A100/H100), vLLM supports tensor parallelism: splitting the model's matrices across multiple GPUs that cooperate on each forward pass. This lets you serve large models on multi-GPU nodes. llama.cpp supports multi-GPU too (via its own scheme), but vLLM's tensor parallelism is tuned for the production serving regime where it most matters.
vLLM exposes an OpenAI-compatible HTTP API. Any client written against the OpenAI SDK — and there are many — works against vLLM with only a base-URL change. This is a significant adoption advantage: you do not rewrite your application to switch from an OpenAI model to a self-hosted open model.
Independent benchmarks reinforce the throughput story. Red Hat published a llama.cpp-vs-vLLM comparison (part of their OpenShift AI / InstructLab work) that characterizes the trade-off: llama.cpp offers broad deployability and lower-overhead single-stream performance, while vLLM delivers substantially higher aggregate throughput under concurrent multi-user load on GPUs. This matches the "two tools, two jobs" framing — neither dominates; each wins in its regime.
llama.cpp is the air-gap standard. Above vLLM sits a high-end tier for the most demanding serving workloads.
llama.cpp makes no network calls and collects no telemetry. It is a single static binary you can drop onto a machine that has never touched the internet. This is not incidental — it is why llama.cpp is the default for air-gapped and sensitive-domain deployments (Pillar 7, Modules FT21–FT22). For HIPAA, government, and classified environments, a server that phones home is a non-starter. llama.cpp's posture — one binary, no network dependency, no telemetry — satisfies the procurement and security requirements those environments impose.
vLLM's posture is also clean for self-hosted serving (it does not force telemetry on you), but it is a Python application with a dependency tree, which is a heavier assurance surface than a single static binary. For the strictest air-gap requirements, the single binary wins on auditability.
Above vLLM sits a tier of engines for the most demanding production workloads:
TensorRT-LLM (NVIDIA) is the low-latency, high-throughput engine built on NVIDIA's TensorRT. It is the choice when you are on NVIDIA hardware, you need the absolute lowest latency and highest throughput, and you are willing to do the extra compilation/optimization work (building engines, tuning kernels) that TensorRT requires. It is the reference for NVIDIA-optimized serving but adds operational complexity.
SGLang is a newer engine (originating from research, arXiv:2312.07104) that adds structured generation and program-level optimizations (RadixAttention for prefix caching, structured output constraints). It competes with vLLM on raw throughput and can exceed it on workloads with shared prefixes (e.g., many requests sharing a long system prompt) or structured-output requirements. For teams whose workload fits SGLang's strengths — heavy prefix reuse, JSON/constrained generation — it is worth evaluating alongside vLLM.
The tiering: llama.cpp for breadth and air-gap; vLLM for the production GPU default; SGLang and TensorRT-LLM for the high end where you need to squeeze the last margins of latency and throughput. Most teams start with vLLM and only move up when they have a measured reason to.
GGUF will not run in vLLM; AWQ will not run in llama.cpp. The serving engine (decided by deployment context) dictates the quant format. Decide the server first, then export to its format. A team that quantizes to AWQ and then needs to deploy to an air-gapped CPU box has to re-export.
llama.cpp can serve concurrent requests, but it is not optimized for the PagedAttention + continuous-batching regime that vLLM is built for. If you have GPUs and real concurrent traffic, vLLM's throughput advantage is large. Reaching for llama.cpp out of familiarity caps your capacity.
vLLM is GPU-oriented. On a CPU-only machine or a strictly air-gapped box with heterogeneous hardware, llama.cpp is the tool. vLLM on CPU is not its design point.
TensorRT-LLM's extra compilation and tuning complexity is only justified when you have measured that vLLM is your bottleneck and NVIDIA-optimized kernels will recover meaningful margin. Start with vLLM. Move up with data.
| Term | Definition |
|---|---|
| llama.cpp | C/C++ single-binary inference engine. Maximal hardware breadth (CPU/Metal/CUDA/ROCm/Vulkan). Air-gap default. The engine under Ollama and LM Studio. |
| vLLM | Python-based GPU serving engine. PagedAttention, continuous batching, tensor parallelism. The production GPU default with an OpenAI-compatible API. |
| GGUF | llama.cpp's native container format. Bundles weights, tokenizer, metadata in one file. Uses k-quants (Q4_K_M, Q5_K_M, Q8_0). |
| k-quants | llama.cpp's block-wise quantization schemes (the K variants). Adapt precision per weight block; outperform naive quantization at the same bit width. |
| AWQ / GPTQ | GPU-native quantization formats served by vLLM. AWQ is activation-aware; GPTQ is a post-training method. Both require GPU kernels. |
| FP8 | Native 8-bit float format on newer NVIDIA GPUs (Hopper, Blackwell). Served by vLLM on supported hardware for low-loss compression. |
| PagedAttention | vLLM's KV-cache management: cache as virtual memory, allocated in fixed-size blocks on demand. Eliminates fragmentation; raises concurrent-request capacity. (Kwon et al., SOSP 2023.) |
| Continuous batching | Admitting new requests into a batch at the iteration/token level as slots free, rather than waiting for the batch to complete. Keeps the GPU saturated. |
| Tensor parallelism | Splitting a model's matrices across multiple GPUs that cooperate per forward pass. Lets vLLM serve models too large for one GPU. |
| TensorRT-LLM | NVIDIA's low-latency, high-throughput engine built on TensorRT. The high-end NVIDIA-optimized tier above vLLM. |
| SGLang | A newer engine with RadixAttention (prefix caching) and structured-generation optimizations. Competes with vLLM; excels on shared-prefix and constrained-output workloads. |
See 07-lab-spec.md. The "Serve Both Ways" lab: export a model to GGUF and serve it with llama.cpp's server, then serve the same model with vLLM. Compare the setup, the API, and the throughput characteristics. You will feel why each tool has the deployment niche it has.
github.com/ggerganov/llama.cpp. The llama.cpp project, documentation, and the GGUF format spec.github.com/vllm-project/vllm. Documentation on continuous batching, tensor parallelism, and the OpenAI-compatible API.# Module FTDD-09 — llama.cpp + vLLM **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-09 — llama.cpp + vLLM **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT19 (Quantization), FT20 (Serving) --- ## Learning Objectives After completing this module, you will be able to: 1. Distinguish llama.cpp and vLLM by deployment context — llama.cpp for hardware breadth and air-gap, vLLM for production GPU throughput — and choose correctly for a given environment. 2. Explain why quant formats follow the server: GGUF for llama.cpp, AWQ/GPTQ/FP8 for vLLM. Predict which quant you can serve in which engine. 3. Describe vLLM's throughput advantages (PagedAttention, continuous batching, tensor parallelism) and why they make it the production GPU default with an OpenAI-compatible API. 4. State the no-telemetry posture of each and why llama.cpp is the air-gap standard for sensitive domains (Pillar 7), and place SGLang and TensorRT-LLM as the high-end tier above vLLM. --- # 9.1 — Two Tools, Two Jobs *Stop treating llama.cpp and vLLM as rivals. They are complementary tools for different jobs. The deployment context decides which you reach for.* ## llama.cpp: the universal single-binary server llama.cpp is a C/C++ inference engine, distributed as (effectively) a single binary with no Python runtime dependency. Its defining property is **hardware breadth**: the same binary runs on CPU, Apple Metal, NVIDIA CUDA, AMD ROCm, and Vulkan backends. If a machine exists, llama.cpp can probably run a model on it. You compile once per platform (or download a prebuilt release) and you have a server. This breadth is why llama.cpp is the tool for the long tail of deployment targets: the developer laptop, the edge box, the air-gapped server with whatever hardware happens to be in it, the M-series Mac doing local inference. It is also the engine underneath Ollama and LM Studio — most "local LLM" tools are llama.cpp with a wrapper. When you serve with llama.cpp's built-in server, you get an OpenAI-compatible HTTP API on whatever port you choose, on whatever hardware you have. ## vLLM: the production GPU throughput engine vLLM is a Python-based inference engine built for one thing: maximum throughput on GPUs in a serving context. Where llama.cpp optimizes for breadth and portability, vLLM optimizes for requests-per-second on NVIDIA (and increasingly AMD) GPUs in a datacenter. Its defining properties are PagedAttention (which we cover in 9.3) and continuous batching, together with multi-GPU tensor parallelism for models too large for one GPU. vLLM is what you reach for when you have real traffic — a production endpoint serving many concurrent users — and GPUs to serve it on. It exposes an OpenAI-compatible API out of the box, which means anything written against the OpenAI SDK works against vLLM unchanged. For most cloud-deployed open-model services, vLLM is the default. ## The decision is about context, not preference The mistake is treating them as competing options and picking a favorite. They are not. A team that ships a local desktop assistant uses llama.cpp (the user's laptop may not have a GPU; the binary must be self-contained). A team that ships a cloud API with hundreds of concurrent users uses vLLM (throughput is the constraint, and there are GPUs). Many teams use both: llama.cpp for the local/edge component and vLLM for the cloud service. The deployment context — what hardware, what network, what traffic profile — is the entire decision. --- # 9.2 — The Quant-Format Compatibility *The quant format you choose is downstream of the serving engine you choose. This is non-negotiable.* ## GGUF is llama.cpp's format GGUF (GPT-Generated Unified Format) is llama.cpp's native container. It bundles the model weights, tokenizer, and metadata in a single file. llama.cpp's quantization produces k-quants — named like Q4_K_M, Q5_K_M, Q8_0 — which are llama.cpp-specific precision schemes designed to preserve quality at low bit-widths across the heterogeneous hardware llama.cpp targets. When you download a model from HuggingFace to run in Ollama or llama.cpp, you download a GGUF. The k-quants matter because they are the reason llama.cpp can run a usable 7B model on a consumer laptop. Q4_K_M is the widely-recommended default — a good quality/size trade-off for most models. The K-quants use block-wise scaling that adapts precision to the importance of different weight groups, which is why they outperform naive round-to-nearest quantization at the same bit width. ## AWQ, GPTQ, and FP8 are vLLM's formats vLLM serves GPU-native quantization formats: AWQ (Activation-aware Weight Quantization), GPTQ (a post-training quantization method), and increasingly FP8 (the native 8-bit float format on newer NVIDIA GPUs like Hopper and Blackwell). These formats are designed for GPU execution kernels — they are optimized for the memory bandwidth and compute patterns of CUDA/ROCm, not for CPU portability. The critical compatibility rule: **you cannot serve a GGUF in vLLM, and you cannot serve an AWQ or GPTQ in llama.cpp.** The formats are tied to their engines' kernels. This means the quant decision is not independent of the serving decision. You choose the server first (based on deployment context, per 9.1), and the quant format follows. ## The practical workflow This creates two distinct export paths after fine-tuning (Layer 4 of the Steering Stack). If you are deploying to llama.cpp, you export to GGUF (using llama.cpp's `convert` and `quantize` tools, or the `llama-cpp-python` converter). If you are deploying to vLLM, you export to AWQ or GPTQ (using AutoAWQ or AutoGPTQ), or you serve in FP16/BF16 if you have the VRAM and want zero quantization loss, or FP8 on supported hardware. A team that ships both local and cloud will maintain two export artifacts from the same trained weights: a GGUF for the edge and an AWQ (or FP16) for the cloud. --- # 9.3 — Why vLLM Wins Production Throughput *PagedAttention and continuous batching are the two innovations that make vLLM the production GPU default.* ## PagedAttention: the KV cache as virtual memory During generation, a transformer must store the keys and values for every token it has processed so far — the KV cache. In a naive serving implementation, this cache is pre-allocated as a contiguous block per request, sized for the maximum sequence length. The problem: real requests have variable lengths, and most do not use their full allocation. The result is severe memory fragmentation and waste — a large fraction of GPU memory sits allocated but unused, capping how many concurrent requests you can serve. PagedAttention, introduced in the vLLM paper (Kwon et al., arXiv:2309.06180, SOSP 2023), manages the KV cache like an operating system manages virtual memory. The cache is broken into fixed-size blocks (pages), and each request's cache is a mapping of logical-to-physical blocks, allocated on demand. This eliminates the fragmentation: memory is only used for tokens actually generated. The practical effect is a dramatic increase in how many concurrent requests fit on a GPU, which directly increases throughput. ## Continuous batching: admit new requests mid-generation Traditional batching (static batching) waits to fill a batch, processes it, then starts the next. If one request in the batch is much longer than the others, the short requests finish early and their slots sit idle until the long one finishes. Throughput suffers. Continuous batching (sometimes called iteration-level or dynamic batching) admits new requests into the batch as soon as a slot frees — at the token/iteration level, not the batch level. As soon as one request finishes generation, a waiting request takes its place immediately. Combined with PagedAttention's efficient memory use, this means the GPU stays saturated: there is always work to do, and there is always memory for new requests. This is the second major reason vLLM achieves high throughput under concurrent load. ## Multi-GPU tensor parallelism For models too large for a single GPU (e.g., a 70B model in FP16 needs ~140GB, beyond one A100/H100), vLLM supports tensor parallelism: splitting the model's matrices across multiple GPUs that cooperate on each forward pass. This lets you serve large models on multi-GPU nodes. llama.cpp supports multi-GPU too (via its own scheme), but vLLM's tensor parallelism is tuned for the production serving regime where it most matters. ## The OpenAI-compatible API and the Red Hat benchmark vLLM exposes an OpenAI-compatible HTTP API. Any client written against the OpenAI SDK — and there are many — works against vLLM with only a base-URL change. This is a significant adoption advantage: you do not rewrite your application to switch from an OpenAI model to a self-hosted open model. Independent benchmarks reinforce the throughput story. Red Hat published a llama.cpp-vs-vLLM comparison (part of their OpenShift AI / InstructLab work) that characterizes the trade-off: llama.cpp offers broad deployability and lower-overhead single-stream performance, while vLLM delivers substantially higher aggregate throughput under concurrent multi-user load on GPUs. This matches the "two tools, two jobs" framing — neither dominates; each wins in its regime. --- # 9.4 — The Air-Gap Default and the High-End Tier *llama.cpp is the air-gap standard. Above vLLM sits a high-end tier for the most demanding serving workloads.* ## llama.cpp's no-telemetry posture llama.cpp makes no network calls and collects no telemetry. It is a single static binary you can drop onto a machine that has never touched the internet. This is not incidental — it is why llama.cpp is the default for air-gapped and sensitive-domain deployments (Pillar 7, Modules FT21–FT22). For HIPAA, government, and classified environments, a server that phones home is a non-starter. llama.cpp's posture — one binary, no network dependency, no telemetry — satisfies the procurement and security requirements those environments impose. vLLM's posture is also clean for self-hosted serving (it does not force telemetry on you), but it is a Python application with a dependency tree, which is a heavier assurance surface than a single static binary. For the strictest air-gap requirements, the single binary wins on auditability. ## SGLang and TensorRT-LLM: the high-end tier Above vLLM sits a tier of engines for the most demanding production workloads: - **TensorRT-LLM** (NVIDIA) is the low-latency, high-throughput engine built on NVIDIA's TensorRT. It is the choice when you are on NVIDIA hardware, you need the absolute lowest latency and highest throughput, and you are willing to do the extra compilation/optimization work (building engines, tuning kernels) that TensorRT requires. It is the reference for NVIDIA-optimized serving but adds operational complexity. - **SGLang** is a newer engine (originating from research, arXiv:2312.07104) that adds structured generation and program-level optimizations (RadixAttention for prefix caching, structured output constraints). It competes with vLLM on raw throughput and can exceed it on workloads with shared prefixes (e.g., many requests sharing a long system prompt) or structured-output requirements. For teams whose workload fits SGLang's strengths — heavy prefix reuse, JSON/constrained generation — it is worth evaluating alongside vLLM. The tiering: llama.cpp for breadth and air-gap; vLLM for the production GPU default; SGLang and TensorRT-LLM for the high end where you need to squeeze the last margins of latency and throughput. Most teams start with vLLM and only move up when they have a measured reason to. --- ## Anti-Patterns ### Choosing the quant before the server GGUF will not run in vLLM; AWQ will not run in llama.cpp. The serving engine (decided by deployment context) dictates the quant format. Decide the server first, then export to its format. A team that quantizes to AWQ and then needs to deploy to an air-gapped CPU box has to re-export. ### Using llama.cpp for high-concurrency cloud serving llama.cpp can serve concurrent requests, but it is not optimized for the PagedAttention + continuous-batching regime that vLLM is built for. If you have GPUs and real concurrent traffic, vLLM's throughput advantage is large. Reaching for llama.cpp out of familiarity caps your capacity. ### Using vLLM for an air-gapped CPU-only box vLLM is GPU-oriented. On a CPU-only machine or a strictly air-gapped box with heterogeneous hardware, llama.cpp is the tool. vLLM on CPU is not its design point. ### Reaching for TensorRT-LLM prematurely TensorRT-LLM's extra compilation and tuning complexity is only justified when you have measured that vLLM is your bottleneck and NVIDIA-optimized kernels will recover meaningful margin. Start with vLLM. Move up with data. --- ## Key Terms | Term | Definition | | --- | --- | | **llama.cpp** | C/C++ single-binary inference engine. Maximal hardware breadth (CPU/Metal/CUDA/ROCm/Vulkan). Air-gap default. The engine under Ollama and LM Studio. | | **vLLM** | Python-based GPU serving engine. PagedAttention, continuous batching, tensor parallelism. The production GPU default with an OpenAI-compatible API. | | **GGUF** | llama.cpp's native container format. Bundles weights, tokenizer, metadata in one file. Uses k-quants (Q4_K_M, Q5_K_M, Q8_0). | | **k-quants** | llama.cpp's block-wise quantization schemes (the _K_ variants). Adapt precision per weight block; outperform naive quantization at the same bit width. | | **AWQ / GPTQ** | GPU-native quantization formats served by vLLM. AWQ is activation-aware; GPTQ is a post-training method. Both require GPU kernels. | | **FP8** | Native 8-bit float format on newer NVIDIA GPUs (Hopper, Blackwell). Served by vLLM on supported hardware for low-loss compression. | | **PagedAttention** | vLLM's KV-cache management: cache as virtual memory, allocated in fixed-size blocks on demand. Eliminates fragmentation; raises concurrent-request capacity. (Kwon et al., SOSP 2023.) | | **Continuous batching** | Admitting new requests into a batch at the iteration/token level as slots free, rather than waiting for the batch to complete. Keeps the GPU saturated. | | **Tensor parallelism** | Splitting a model's matrices across multiple GPUs that cooperate per forward pass. Lets vLLM serve models too large for one GPU. | | **TensorRT-LLM** | NVIDIA's low-latency, high-throughput engine built on TensorRT. The high-end NVIDIA-optimized tier above vLLM. | | **SGLang** | A newer engine with RadixAttention (prefix caching) and structured-generation optimizations. Competes with vLLM; excels on shared-prefix and constrained-output workloads. | --- ## Lab Exercise See `07-lab-spec.md`. The "Serve Both Ways" lab: export a model to GGUF and serve it with llama.cpp's server, then serve the same model with vLLM. Compare the setup, the API, and the throughput characteristics. You will feel why each tool has the deployment niche it has. --- ## References 1. **Gerganov et al.** — `github.com/ggerganov/llama.cpp`. The llama.cpp project, documentation, and the GGUF format spec. 2. **Kwon et al. (2023)** — *Efficient Memory Management for Large Language Model Serving with PagedAttention*. arXiv:2309.06180, SOSP 2023. The vLLM paper; the source of PagedAttention. 3. **vLLM project** — `github.com/vllm-project/vllm`. Documentation on continuous batching, tensor parallelism, and the OpenAI-compatible API. 4. **Lin et al. (2023)** — *AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration*. arXiv:2306.00978. The AWQ format served by vLLM. 5. **Frantar et al. (2022)** — *GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers*. arXiv:2210.17323. The GPTQ format. 6. **Zheng et al. (2023)** — *SGLang: Efficient Execution of Structured Language Model Programs*. arXiv:2312.07104. RadixAttention and structured generation. 7. **Red Hat** — *llama.cpp vs vLLM benchmark* (Red Hat OpenShift AI / InstructLab documentation). Independent characterization of the trade-off. 8. **Course 3, Modules FT19–FT20** — Quantization and serving. The Layer 4 context this deep-dive extends.