llama.cpp + vLLM
Module FTDD-09 · Course 3 — LLM Fine-Tuning Masterclass
45 minutes · 4 sub-sections: Two Tools, Two Jobs · Quant Compatibility · Why vLLM Wins · Air-Gap + High-End Tier
The serving deep dive. How you ship the model you just trained.
Deep-Dives
The question every fine-tuner faces
You trained a model. How do you serve it?
Two answers every senior engineer must know:
llama.cpp
Single C/C++ binary. Max hardware breadth. Air-gap default.
vLLM
Python GPU engine. Max production throughput. Cloud default.
Not competitors — complementary. The deployment context decides.
llama.cpp — the universal binary
| Property | Why it matters |
|---|
| Single C/C++ binary | No Python runtime. Drop on any machine. |
| CPU · Metal · CUDA · ROCm · Vulkan | If hardware exists, llama.cpp runs on it. |
| OpenAI-compatible HTTP server | Built-in. Whatever port, whatever hardware. |
| Under Ollama & LM Studio | Most "local LLM" tools are llama.cpp + wrapper. |
Defining property: hardware breadth. The long tail of targets: laptops, edge boxes, air-gapped servers, M-series Macs.
vLLM — the production GPU engine
| Property | Why it matters |
|---|
| Python-based, GPU-focused | Optimized for requests/sec on NVIDIA/AMD GPUs. |
| PagedAttention | KV cache as virtual memory. No fragmentation. |
| Continuous batching | Admit new requests mid-generation. |
| Tensor parallelism | Serve models too large for one GPU. |
| OpenAI-compatible API | Any OpenAI SDK client works, base-URL change only. |
Defining property: throughput under concurrent load. The default for cloud-deployed open-model services.
Two tools, two jobs — the decision
Reach for llama.cpp when
- Hardware is CPU / Mac / heterogeneous
- Air-gapped / sensitive domain (Pillar 7)
- You need a single self-contained binary
- Local / edge / desktop deployment
Reach for vLLM when
- GPU datacenter, concurrent users
- Throughput is the constraint
- You want the OpenAI-compatible serving API
- Cloud API with real traffic
Many teams use both: llama.cpp for local/edge, vLLM for the cloud service.
Quant formats follow the server
You CANNOT cross-serve. GGUF runs in llama.cpp. AWQ/GPTQ/FP8 run in vLLM. The quant choice is downstream of the serving choice.
| Server | Formats | Default quant |
|---|
| llama.cpp | GGUF (k-quants) | Q4_K_M (quality/size sweet spot) |
| vLLM | AWQ · GPTQ · FP8 · FP16/BF16 | AWQ, or FP16 if VRAM allows |
Two export paths from the same trained weights: a GGUF for the edge, an AWQ (or FP16) for the cloud.
Why vLLM wins production throughput
PagedAttention (SOSP 2023)
KV cache managed like virtual memory — fixed-size blocks, allocated on demand.
Eliminates fragmentation. Many more concurrent requests per GPU.
Continuous batching
Admit new requests at the iteration/token level as slots free.
The GPU stays saturated. No idle slots waiting for long requests.
Together: high aggregate throughput, fully utilized GPU memory. (Kwon et al., arXiv:2309.06180.)
The Red Hat benchmark
Independent characterization of the llama.cpp vs vLLM trade-off (Red Hat OpenShift AI / InstructLab).
| Engine | Wins on |
|---|
| llama.cpp | Broad deployability; lower-overhead single-stream performance |
| vLLM | Substantially higher aggregate throughput under concurrent multi-user load on GPUs |
Neither dominates. Each wins in its regime. This matches "two tools, two jobs."
llama.cpp — the air-gap default
No network calls. No telemetry. One static binary. Drop on a machine that has never touched the internet.
Why this matters:
- HIPAA / government / classified: a server that phones home is a non-starter.
- Single binary = minimal assurance surface for security review.
- The standard for Pillar 7 (sensitive domains, FT21–FT22).
vLLM is also clean for self-hosting but is a Python app with a dependency tree — heavier to audit than one binary.
The serving tiers
TENSORRT-LLM (NVIDIA) · lowest latency, NVIDIA-only, extra compilation
↑
SGLang · RadixAttention prefix caching, structured generation
↑
vLLM · production GPU default — START HERE
↑
llama.cpp · universal single binary, air-gap default
Most teams start at vLLM. Move up to SGLang/TensorRT-LLM only with measured reason that vLLM is the bottleneck.
Anti-patterns
Choosing the quant before the server. GGUF won't run in vLLM; AWQ won't run in llama.cpp. Decide the server first (deployment context), then export to its format.
llama.cpp for high-concurrency cloud. Not built for PagedAttention + continuous batching. If you have GPUs and real traffic, vLLM's throughput advantage is large.
Reaching for TensorRT-LLM prematurely. Its extra complexity is justified only when vLLM is a measured bottleneck. Start with vLLM. Move up with data.
What you can now do
- Distinguish llama.cpp and vLLM by deployment context and choose correctly.
- Predict which quant formats run in which engine (GGUF vs AWQ/GPTQ/FP8).
- Explain vLLM's throughput advantages (PagedAttention, continuous batching, tensor parallelism).
- State why llama.cpp is the air-gap default, and when to reach for SGLang/TensorRT-LLM.
Next: FTDD-10 — distilabel (the synthetic data pipeline framework)