Lab Specification — Module FTDD-09: llama.cpp + vLLM

Course: Course 3 — LLM Fine-Tuning Masterclass Module: FTDD-09 — llama.cpp + vLLM Duration: ~45 minutes Environment: Python 3.11+. GPU recommended (NVIDIA, 16GB+ VRAM) OR Apple Silicon (M-series) for the llama.cpp half. For the vLLM half, a CUDA GPU is required — Colab T4/A100 works. ~8GB free disk.

Note: This lab has two halves that may run in different environments. The llama.cpp half runs on CPU/Mac/GPU. The vLLM half requires a CUDA GPU. If you lack a local CUDA GPU, run the vLLM half on Google Colab.

Learning objectives

By the end of this lab you will have:

Exported a model to GGUF and served it with llama.cpp's built-in server — felt the single-binary, hardware-breadth deployment.
Served the same model with vLLM — felt the production GPU serving setup with the OpenAI-compatible API.
Observed the quant-format constraint directly: the GGUF serves in llama.cpp, the original/AWQ weights serve in vLLM, and you cannot cross them.
Compared the two serving experiences and stated, for a given deployment context, which engine you would choose and why.

Phase 0 — Environment setup (5 min)

You need a small model available in both ecosystems. Qwen2.5-1.5B (or Qwen3-1.7B) works well — it fits on modest hardware and has both GGUF and HF-weight versions.

# Two environments. Create both if your machine supports them.

# For the llama.cpp half:
# Install llama.cpp — easiest is via pip wrapper, or download a prebuilt binary
pip install llama-cpp-python[server]

# For the vLLM half (CUDA GPU required):
pip install vllm

Phase 1 — Serve with llama.cpp (GGUF) (12 min)

Download a GGUF of your chosen model from HuggingFace (search for the model name + GGUF). For Qwen2.5-1.5B, a Q4_K_M GGUF is ~1GB.

# Download a GGUF (example path — use the actual HF repo and file)
# e.g., huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf

# Serve with llama.cpp's server (via llama-cpp-python)
python -m llama_cpp.server \
  --model ./qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 --port 8000 \
  --n_ctx 2048

In another terminal, test the OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-1.5b",
    "messages": [{"role": "user", "content": "What is 15 * 17? Think step by step."}],
    "max_tokens": 200
  }'

Record: that llama.cpp served a GGUF and responded to an OpenAI-compatible request. Note the ease: one binary, one file, one port.

What just happened: You deployed a model with a single binary, no Python serving runtime, on whatever hardware you have. This is llama.cpp's defining property — hardware breadth and deployment simplicity. If your machine is air-gapped, this exact setup works unchanged.

Phase 2 — Serve with vLLM (12 min)

# Serve the same model family from HF weights (CUDA GPU required)
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --host 0.0.0.0 --port 8001 \
  --max-model-len 2048

In another terminal:

curl http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "What is 15 * 17? Think step by step."}],
    "max_tokens": 200
  }'

Record: that vLLM served the HF weights and responded to the same OpenAI-compatible request shape. Note the differences: vLLM loads from the HF weight repo, reports GPU memory and KV-cache allocation, and is built around the GPU.

Phase 3 — Feel the quant-format constraint (8 min)

Try the forbidden combinations and observe the errors:

# Attempt 1: serve a GGUF in vLLM — should fail
python -m vllm.entrypoints.openai.api_server --model ./qwen2.5-1.5b-instruct-q4_k_m.gguf

# Attempt 2 (conceptual): you cannot point llama.cpp at an AWQ either.
# The formats are tied to their engines' kernels.

Record: the error vLLM gives when handed a GGUF. This is the hard constraint made visible — you cannot cross-serve.

Phase 4 — The deployment decision, in your own words (5 min)

No code. Write 3–5 sentences answering:

For each half, what was the setup experience? Which felt lighter? Which felt built for throughput?
If you were deploying to an air-gapped government server with a single CPU, which engine and which format? Why?
If you were deploying a cloud API with 200 concurrent users on 4x A100 GPUs, which engine and format? Why? What would you gain by moving up to TensorRT-LLM, and why might you not bother?

Deliverables

Submit ftdd09-lab-report.md:

Phase 1: confirmation llama.cpp served the GGUF; the curl response (note the one-binary simplicity)
Phase 2: confirmation vLLM served the HF weights; the curl response (note the GPU/throughput orientation)
Phase 3: the error when you tried to cross-serve (GGUF in vLLM)
Phase 4: your 3–5 sentence deployment decision

Solution key

Phase 1: llama.cpp serves the GGUF on port 8000 and returns a valid chat completion. The setup is one binary + one file + one command — this is the hardware-breadth/air-gap property. Works on CPU, Mac, or GPU without changing the binary.
Phase 2: vLLM serves the HF weights on port 8001 and returns a valid chat completion with the same request shape (OpenAI-compatible). The logs show GPU memory allocation and KV-cache block setup (PagedAttention in action). This is the production GPU throughput engine.
Phase 3: vLLM rejects the GGUF — it expects HF-format weights or an AWQ/GPTQ/FP8 checkpoint, not a GGUF container. The error is the quant-format constraint made visible: formats follow the server.
Phase 4: a correct decision names (a) llama.cpp felt lighter and more portable; vLLM felt GPU-and-throughput-oriented; (b) for the air-gapped government CPU box: llama.cpp with a GGUF (Q4_K_M) — single binary, no telemetry, CPU support; (c) for the 200-concurrent-user cloud API on GPUs: vLLM with AWQ or FP16 — PagedAttention + continuous batching maximize throughput. Moving to TensorRT-LLM would gain latency/throughput margins at the cost of compilation complexity; justified only if vLLM is a measured bottleneck.

Notes on environment

vLLM requires CUDA (or a supported ROCm setup). On a Mac or CPU-only box, the vLLM half will not run locally — use Colab or a cloud GPU for that half.
llama.cpp runs anywhere. If you only have one environment, prioritize the half that runs there and read the other half's commands as reference.
Model choice: use a 1.5B–3B model to keep VRAM and download size manageable. The principles are identical for larger models.

Stretch goals

Stress-test vLLM's batching. Write a script that fires 20 concurrent requests at the vLLM server and measure aggregate throughput (requests/sec). Then do the same against llama.cpp and compare. This makes the PagedAttention + continuous-batching advantage tangible.
Export your own GGUF. Take a small model you fine-tuned (or a base) and run llama.cpp's convert + quantize tools to produce a Q4_K_M GGUF. This is the actual Layer 4 export path you would use in production. (Connects to Module FT19.)
Try SGLang. If you have a workload with a long shared system prompt (prefix reuse), serve it with SGLang and compare against vLLM. Observe RadixAttention's prefix-cache benefit when many requests share the same prefix. (Sets up awareness of the high-end tier.)

# Lab Specification — Module FTDD-09: llama.cpp + vLLM **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-09 — llama.cpp + vLLM **Duration**: ~45 minutes **Environment**: Python 3.11+. **GPU recommended** (NVIDIA, 16GB+ VRAM) OR Apple Silicon (M-series) for the llama.cpp half. For the vLLM half, a CUDA GPU is required — Colab T4/A100 works. ~8GB free disk. > **Note**: This lab has two halves that may run in different environments. The llama.cpp half runs on CPU/Mac/GPU. The vLLM half requires a CUDA GPU. If you lack a local CUDA GPU, run the vLLM half on Google Colab. --- ## Learning objectives By the end of this lab you will have: 1. **Exported a model to GGUF** and served it with llama.cpp's built-in server — felt the single-binary, hardware-breadth deployment. 2. **Served the same model with vLLM** — felt the production GPU serving setup with the OpenAI-compatible API. 3. **Observed the quant-format constraint directly**: the GGUF serves in llama.cpp, the original/AWQ weights serve in vLLM, and you cannot cross them. 4. **Compared the two serving experiences** and stated, for a given deployment context, which engine you would choose and why. --- ## Phase 0 — Environment setup (5 min) You need a small model available in both ecosystems. Qwen2.5-1.5B (or Qwen3-1.7B) works well — it fits on modest hardware and has both GGUF and HF-weight versions. ```bash # Two environments. Create both if your machine supports them. # For the llama.cpp half: # Install llama.cpp — easiest is via pip wrapper, or download a prebuilt binary pip install llama-cpp-python[server] # For the vLLM half (CUDA GPU required): pip install vllm ``` --- ## Phase 1 — Serve with llama.cpp (GGUF) (12 min) Download a GGUF of your chosen model from HuggingFace (search for the model name + GGUF). For Qwen2.5-1.5B, a Q4_K_M GGUF is ~1GB. ```bash # Download a GGUF (example path — use the actual HF repo and file) # e.g., huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf # Serve with llama.cpp's server (via llama-cpp-python) python -m llama_cpp.server \ --model ./qwen2.5-1.5b-instruct-q4_k_m.gguf \ --host 0.0.0.0 --port 8000 \ --n_ctx 2048 ``` In another terminal, test the OpenAI-compatible API: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen2.5-1.5b", "messages": [{"role": "user", "content": "What is 15 * 17? Think step by step."}], "max_tokens": 200 }' ``` **Record**: that llama.cpp served a GGUF and responded to an OpenAI-compatible request. Note the ease: one binary, one file, one port. > **What just happened:** You deployed a model with a single binary, no Python serving runtime, on whatever hardware you have. This is llama.cpp's defining property — hardware breadth and deployment simplicity. If your machine is air-gapped, this exact setup works unchanged. --- ## Phase 2 — Serve with vLLM (12 min) ```bash # Serve the same model family from HF weights (CUDA GPU required) python -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen2.5-1.5B-Instruct \ --host 0.0.0.0 --port 8001 \ --max-model-len 2048 ``` In another terminal: ```bash curl http://localhost:8001/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "What is 15 * 17? Think step by step."}], "max_tokens": 200 }' ``` **Record**: that vLLM served the HF weights and responded to the same OpenAI-compatible request shape. Note the differences: vLLM loads from the HF weight repo, reports GPU memory and KV-cache allocation, and is built around the GPU. --- ## Phase 3 — Feel the quant-format constraint (8 min) Try the forbidden combinations and observe the errors: ```bash # Attempt 1: serve a GGUF in vLLM — should fail python -m vllm.entrypoints.openai.api_server --model ./qwen2.5-1.5b-instruct-q4_k_m.gguf # Attempt 2 (conceptual): you cannot point llama.cpp at an AWQ either. # The formats are tied to their engines' kernels. ``` **Record**: the error vLLM gives when handed a GGUF. This is the hard constraint made visible — you cannot cross-serve. --- ## Phase 4 — The deployment decision, in your own words (5 min) No code. Write 3–5 sentences answering: 1. For each half, what was the setup experience? Which felt lighter? Which felt built for throughput? 2. If you were deploying to an air-gapped government server with a single CPU, which engine and which format? Why? 3. If you were deploying a cloud API with 200 concurrent users on 4x A100 GPUs, which engine and format? Why? What would you gain by moving up to TensorRT-LLM, and why might you not bother? --- ## Deliverables Submit `ftdd09-lab-report.md`: - [ ] Phase 1: confirmation llama.cpp served the GGUF; the curl response (note the one-binary simplicity) - [ ] Phase 2: confirmation vLLM served the HF weights; the curl response (note the GPU/throughput orientation) - [ ] Phase 3: the error when you tried to cross-serve (GGUF in vLLM) - [ ] Phase 4: your 3–5 sentence deployment decision --- ## Solution key - **Phase 1**: llama.cpp serves the GGUF on port 8000 and returns a valid chat completion. The setup is one binary + one file + one command — this is the hardware-breadth/air-gap property. Works on CPU, Mac, or GPU without changing the binary. - **Phase 2**: vLLM serves the HF weights on port 8001 and returns a valid chat completion with the same request shape (OpenAI-compatible). The logs show GPU memory allocation and KV-cache block setup (PagedAttention in action). This is the production GPU throughput engine. - **Phase 3**: vLLM rejects the GGUF — it expects HF-format weights or an AWQ/GPTQ/FP8 checkpoint, not a GGUF container. The error is the quant-format constraint made visible: formats follow the server. - **Phase 4**: a correct decision names (a) llama.cpp felt lighter and more portable; vLLM felt GPU-and-throughput-oriented; (b) for the air-gapped government CPU box: llama.cpp with a GGUF (Q4_K_M) — single binary, no telemetry, CPU support; (c) for the 200-concurrent-user cloud API on GPUs: vLLM with AWQ or FP16 — PagedAttention + continuous batching maximize throughput. Moving to TensorRT-LLM would gain latency/throughput margins at the cost of compilation complexity; justified only if vLLM is a measured bottleneck. --- ## Notes on environment - **vLLM requires CUDA** (or a supported ROCm setup). On a Mac or CPU-only box, the vLLM half will not run locally — use Colab or a cloud GPU for that half. - **llama.cpp runs anywhere.** If you only have one environment, prioritize the half that runs there and read the other half's commands as reference. - **Model choice**: use a 1.5B–3B model to keep VRAM and download size manageable. The principles are identical for larger models. --- ## Stretch goals 1. **Stress-test vLLM's batching.** Write a script that fires 20 concurrent requests at the vLLM server and measure aggregate throughput (requests/sec). Then do the same against llama.cpp and compare. This makes the PagedAttention + continuous-batching advantage tangible. 2. **Export your own GGUF.** Take a small model you fine-tuned (or a base) and run llama.cpp's `convert` + `quantize` tools to produce a Q4_K_M GGUF. This is the actual Layer 4 export path you would use in production. (Connects to Module FT19.) 3. **Try SGLang.** If you have a workload with a long shared system prompt (prefix reuse), serve it with SGLang and compare against vLLM. Observe RadixAttention's prefix-cache benefit when many requests share the same prefix. (Sets up awareness of the high-end tier.)