vLLM Framework and Inference Engine Design

Post 04 ended on two facts: decoding is limited by memory traffic, and the KV cache is what fills the GPU. This post is about vLLM, the open-source inference engine built around those two facts. Its central ideas are PagedAttention, which stores the KV cache in small blocks the way an operating system stores memory in pages, and continuous batching, which lets requests join and leave the running batch at every step.

The fifth post covers three topics:

• Why vLLM is especially good for high-concurrency inference
• vLLM's memory management strategy
• How users and applications interact with vLLM

The numbers use the same setup as post 04: Llama-2-7B in FP16 on an A100-class GPU, with 512 KiB of KV cache per token.

#Why is vLLM especially good for high-concurrency inference?

#Why is batching multiple requests difficult for traditional inference engines?

Post 04 showed why we want large batches. A decode step reads every weight once, so serving many sequences in the same step makes each read do many times the work. The difficulty is that requests are not alike: they arrive at different times, have prompts of different lengths, and stop after different numbers of tokens.

A traditional engine runs static batches. It pads every prompt to the longest one, runs the whole batch until its longest output is done, and only then takes new requests:

The useful fraction of a static batch, for prompts of length p_i and outputs of length o_i:

useful fraction = Σ (p_i + o_i)  ÷  ( B × (max p + max o) )

prompts [3, 6, 2, 5], outputs [4, 2, 7, 3]:   32 ÷ (4 × 13) = 32 ÷ 52 ≈ 62%

Real traffic is much worse than this toy example. One 500-token answer in a batch of 50-token answers keeps the whole batch busy ten times longer than most of it needs.

Memory is the second problem. A contiguous KV cache can't grow in place, so each request is given a region sized for the longest sequence it might reach. That wastes memory in three ways:

• Reservation. Slots held for tokens that haven't been generated yet.
• Internal fragmentation. Slots reserved for a maximum length the request never reaches.
• External fragmentation. Gaps between regions that are too small for the next request.

The vLLM paper measured that existing systems used only 20–38% of their KV cache memory for real token states. Since the cache decides how many requests fit at once, that waste directly caps the batch size, and with it the throughput.

#What is the essence of vLLM's PagedAttention?

Store each sequence's KV cache in small, fixed-size blocks that can sit anywhere in GPU memory, and keep a block table per sequence that records where each block is. That is how an operating system gives every process the illusion of one contiguous memory while placing its pages anywhere in RAM.

Finding a token's key and value is simple arithmetic. With a block size B:

token at position t    →  logical block  j = ⌊t / B⌋        slot in the block  r = t mod B
logical block j        →  physical block p = block_table[j]
its key and its value  =  K_cache[p][r]  and  V_cache[p][r]       (one pool per layer)

blocks for T tokens    =  ⌈T / B⌉                wasted slots per sequence: fewer than B

Post 04 drew the bigger picture, with several sequences whose blocks interleave in the same pool. The attention kernel is the only part of the model that has to change: instead of reading one contiguous tensor, it follows the block table.

import torch


def paged_attention(q, block_table, k_cache, v_cache, seq_len, block_size=16):
    # q: (heads, head_dim), the query of the newest token of one sequence
    # block_table: a LongTensor of this sequence's physical block ids
    # k_cache, v_cache: (num_blocks, block_size, heads, head_dim), the shared pool of one layer
    n_blocks = -(-seq_len // block_size)                 # ceil(seq_len / block_size)
    blocks = block_table[:n_blocks]
    k = k_cache[blocks].flatten(0, 1)[:seq_len]          # gather: (seq_len, heads, head_dim)
    v = v_cache[blocks].flatten(0, 1)[:seq_len]
    scores = torch.einsum("hd,thd->ht", q, k) / q.shape[-1] ** 0.5
    weights = torch.softmax(scores, dim=-1)              # (heads, seq_len)
    return torch.einsum("ht,thd->hd", weights, v)        # (heads, head_dim)

What paging buys:

#How does vLLM's scheduling achieve both minimal latency and maximum throughput?

The memory headroom from paging is half of the answer. The other half is continuous batching, also called iteration-level scheduling, an idea from the Orca system: the scheduler re-forms the batch at every decode step, not once per batch.

• A request that finishes leaves the batch at the end of that step, and its blocks go back to the pool straight away.
• A waiting request joins at the next step, as soon as there are free blocks and room in the batch.
• So the GPU never computes empty slots, and a new request waits about one step (tens of milliseconds) instead of a whole batch (seconds).

Each step of the scheduler has a token budget (max_num_batched_tokens) and a cap on the number of sequences (max_num_seqs). It fills them in this order:

1. Running requests first. Each running request gets its next token or, if it is still reading its prompt, the next chunk of the prompt.
2. Make room if needed. If a running request needs a new block and the pool is empty, the scheduler preempts the most recently admitted request. It frees that request's blocks and puts it back in the waiting queue, to be recomputed later.
3. Admit new requests. Waiting requests are admitted in arrival order (or by priority) while the budget, the free blocks and the sequence cap allow. A long prompt is split into chunks to fit the remaining budget, which is called chunked prefill.
4. Run one forward pass. Prompt chunks and decode tokens run together as one batch. Each sequence that has finished its prompt gets a new token, finished requests leave, and the new text is streamed back.

The quantities the scheduler is balancing:

t_step      ≈ ( W_bytes + Σ_i T_i × c_kv ) ÷ BW         post 04: the weights, plus every cached token
throughput  ≈ B ÷ t_step                                B = sequences decoding in the step

TTFT        = queueing delay + prefill time
TPOT        ≈ t_step                                    each running request gets one token per step
latency     = TTFT + (n_out − 1) × TPOT

For Llama-2-7B on an A100, with 1,000 tokens of context per sequence:

Thirty-two times the users cost about twice the step time and give about fifteen times the throughput. Each user's stream slows down, but it stays far faster than anyone can read.

This is how vLLM gets both sides at once:

• Throughput. Continuous batching keeps every slot busy, paging lets more sequences fit, and prefix caching skips prefill work that was already done.
• Latency. Admission at the next step keeps queueing short (time to first token), chunked prefill stops a long prompt from freezing everyone else's stream (time per output token), and blocks are freed the moment a request ends.
• The trade-off is still there. A larger max_num_seqs or max_num_batched_tokens raises throughput but makes each step longer, so every user's tokens come more slowly. Smaller values do the opposite.

#vLLM's memory management strategy

#How is vLLM's KV cache paged?

The pool is set up once, when the engine starts:

1. Load the weights.
2. Profile. Run a forward pass at the largest batch the scheduler allows (max_num_batched_tokens) to measure the peak memory used by activations.
3. Allocate the pool. Everything that is left of gpu_memory_utilization × VRAM becomes KV blocks, allocated as large tensors (one per layer, roughly [2, num_blocks, block_size, kv_heads, head_dim], with the exact layout set by the attention backend).

block_bytes = 2 × L × block_size × H_kv × d × b          K and V, every layer, one block
num_blocks  = ( gpu_memory_utilization × VRAM − weights − peak activations ) ÷ block_bytes

A worked example, for Llama-2-7B on an 80 GB A100 with the defaults:

block_bytes = 2 × 32 × 16 × 32 × 128 × 2 bytes = 8 MiB        (16 tokens × 512 KiB)
budget      = 0.9 × 80 GiB = 72 GiB
pool        = 72 − 12.6 (weights) − about 1.5 (activations) ≈ 58 GiB
num_blocks  ≈ 58 GiB ÷ 8 MiB ≈ 7,400 blocks ≈ 118,000 tokens

Those 118,000 tokens can be shared in any mix: 29 requests at the full 4,096-token context, about 230 requests of 512 tokens, or anything in between. vLLM prints this at startup, as the KV cache size in tokens and the maximum concurrency at the full model length.

After that, the life of a block is short and cheap:

• Writing. When a token's K and V are computed, a small kernel writes them into their slot, block_table[⌊t/B⌋] × B + (t mod B). vLLM calls the list of these per-token slots the slot mapping.
• Growing. When the last block of a sequence is full, the scheduler takes a free block from a free list and appends it to the block table. That is constant-time, and nothing is copied.
• Sharing. Every block has a reference count. With prefix caching (on by default in V1), a full block is also identified by a hash of its own tokens and of every token before it, so a new request whose prompt starts the same way finds the existing blocks and skips that part of the prefill. Post 04 shows two requests sharing a system prompt this way.
• Freeing. When a request ends, each of its blocks loses a reference. A block that reaches zero goes back to the free list, but it keeps its hash, so a later request can still hit it until the space is actually needed. The least recently used blocks are reused first.

#How does it support prompts of arbitrary length? Does it have a sliding window?

Not arbitrary. Every request must fit within max_model_len, which defaults to the context window in the model's configuration. A request whose prompt plus requested output exceeds it is rejected with an error. vLLM also refuses to start if the KV pool can't hold even one sequence of max_model_len tokens, and the error suggests raising gpu_memory_utilization or lowering max_model_len.

Within that limit, three mechanisms make long prompts work:

• Paging. A prompt of T tokens takes ⌈T / 16⌉ blocks, allocated as they are needed. Nothing is reserved for a maximum.
• Chunked prefill. A long prompt is processed in chunks of at most the step's token budget, interleaved with other requests' decode tokens. A 32,000-token prompt never needs activations for 32,000 tokens at once, and nobody else's stream freezes while it is being read.
• Preemption as a last resort. If the pool runs dry part-way through, the most recently admitted request gives its blocks back and tries again later.

admitted only if    prompt_len + max_tokens ≤ max_model_len
prefill steps       ≈ ⌈ prompt_len ÷ chunk ⌉            chunk ≤ max_num_batched_tokens

32,000-token prompt, 2,048-token budget:   about 16 steps, each also carrying everyone else's decodes

Sliding windows. PagedAttention isn't a sliding window, and vLLM doesn't slide a window over a model that wasn't trained with one, because that would change the model's answers. Two different things do exist:

• Models trained with sliding-window attention, such as Mistral 7B v0.1 (a 4,096-token window) or the local layers of Gemma 2 and 3. Each token attends only to the last W tokens, so for those layers only the blocks inside the window are needed, and the older ones can be released. Memory per sequence stops growing.
• Truncation that you ask for. With truncate_prompt_tokens=k in the sampling parameters, vLLM keeps only the last k tokens of the prompt. Otherwise, trimming a long chat history is the client's job.

full attention:      blocks per sequence = ⌈ T / B ⌉                 grows with every token
sliding window W:    blocks per sequence ≤ ⌈ W / B ⌉ + 1             Mistral 7B, W = 4,096:  at most 257

#What operations can cause memory fragmentation, and how does vLLM avoid it?

Post 02 described fragmentation in general: free memory split into pieces too small to use. For a KV cache, these are the operations that cause it:

How vLLM avoids it:

• One pool, allocated once. The KV tensors are created at startup and kept for the engine's lifetime, so admitting or finishing a request never allocates or frees GPU memory. It only moves block ids between lists.
• Every block is the same size. Any free block fits any request, so there is no external fragmentation at all.
• Bounded waste. Only the last block of each sequence can be partly empty: fewer than B slots per sequence, and about B / 2 on average.
• Sharing instead of copying. Reference counts let requests with the same prefix share blocks. The original design also copies a shared block only when a sequence needs to write into it (copy-on-write). The V1 engine shares only full blocks, which are never written again.

contiguous, reserve max_len:    wasted slots = Σ_i ( max_len − T_i )
paged, block size B:            wasted slots = Σ_i ( ⌈T_i / B⌉ × B − T_i )      fewer than N × B

three requests of 300, 1,100 and 600 tokens, max_len = 2,048:
  contiguous:  6,144 slots reserved for 2,000 tokens   →  33% used
  paged:       2,016 slots allocated for 2,000 tokens  →  99% used

#If there are idle and active users simultaneously, how is GPU memory allocated?

vLLM doesn't know about users, only about requests. A chat conversation is a series of separate requests, and each one carries the whole history so far as its prompt. So "active" and "idle" users map onto request states:

So memory follows activity, not users:

• Active requests hold exactly the blocks for the tokens they have.
• An idle user costs nothing guaranteed. Their last turn's blocks sit in the free pool as cached blocks, and they are reused, least recently used first, when someone else needs the space. If the user comes back before that, the new request hits the prefix cache and its time to first token drops.
• When the pool runs out, the most recently admitted running request is preempted. In V1 that means recomputation: its blocks are freed and its prefill is redone later, often partly from the prefix cache. The older V0 engine could also swap blocks out to CPU memory.
• Frequent preemption shows up in the engine's logs and metrics. The usual fixes are more memory for the cache (gpu_memory_utilization), fewer concurrent sequences (max_num_seqs), or more GPUs (tensor parallelism).

blocks in use   = Σ over running requests  ⌈ T_i / B ⌉          × block_bytes
admit request   if   free blocks ≥ ⌈ next chunk ÷ B ⌉   and   running < max_num_seqs

Post 04's lifecycle diagram shows the same pool from the side of a single request.

#How users interact with vLLM

#How is a user's prompt mapped to an internal session?

There is no session. vLLM's unit of work is a request, and it has two front doors:

# Offline: the LLM class, for batch jobs in Python
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=128)

outputs = llm.generate(["The capital of France is", "PagedAttention works by"], params)
for out in outputs:
    print(out.prompt, "->", out.outputs[0].text)

# Online: an OpenAI-compatible HTTP server
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")   # any key works locally
reply = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain the KV cache in one sentence."}],
    max_tokens=64,
)
print(reply.choices[0].message.content)

The path of one request through the server:

1. API server. The front-end process parses the HTTP request, assigns a request id, applies the model's chat template to the messages, and tokenizes the result.
2. Engine core. The tokenized request goes to a separate process as prompt token ids plus sampling parameters.
3. Scheduler. The request enters the waiting queue. Nothing is allocated yet.
4. Admission. When it is scheduled, KV blocks are allocated and recorded in its block table. Blocks found in the prefix cache are reused without computation.
5. Every step. The model runs, one token is sampled for the request and appended to it, and the token is sent back.
6. Detokenizer. Back in the front-end process, token ids are turned into text incrementally, stop strings are checked, and the text is streamed out.

What the internal request carries:

The sampling parameters decide how each new token is drawn from the logits z (post 03 has the details):

p_i = e^(z_i / τ) ÷ Σ_j e^(z_j / τ)            τ = temperature;  τ → 0 is greedy decoding
top_p: sample only from the smallest set of tokens whose probabilities add up to top_p

The whole mapping in one line:

user prompt → request id → chat template → token ids → waiting queue
            → blocks and a block table → batched forward passes → token ids → text, streamed back

Post 06 builds a small version of this same request object for the scheduler project.

#How does vLLM handle cancel, interrupt, or asynchronous responses?

Asynchronous by design. The API server is an asyncio application (FastAPI running on uvicorn). Each HTTP request is a coroutine that waits on its own stream of outputs, while the engine core loops on its own: schedule, run a step, return the outputs. Because the two sides are separate processes, tokenizing and detokenizing overlap with GPU work instead of delaying it.

Streaming. With stream=True, the server sends server-sent events: after each step, a chunk with the new text. The time until the first chunk arrives is the time to first token that the user feels.

Stopping on its own. A request finishes when it samples an end-of-sequence token, produces a stop string or stop token, or reaches max_tokens. The finish_reason in the response says which: stop or length.

Cancelling. If the client disconnects or cancels the stream, the server aborts the request. The engine removes it from the waiting or running queue and frees its blocks, which return to the pool (still cached for prefix reuse). Aborts take effect at a step boundary: a forward pass that is already running on the GPU finishes, so at most one step of work is wasted.

import asyncio

from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")


async def first_words(prompt: str, n_chunks: int = 5) -> str:
    stream = await client.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct", prompt=prompt, max_tokens=256, stream=True
    )
    pieces = []
    async for chunk in stream:
        pieces.append(chunk.choices[0].text)
        if len(pieces) == n_chunks:
            await stream.close()     # closing the connection makes the server abort the request
            break
    return "".join(pieces)


print(asyncio.run(first_words("Once upon a time")))

The states a request moves through:

WAITING → RUNNING → FINISHED_STOPPED        (end-of-sequence token, stop string or stop token)
                  → FINISHED_LENGTH_CAPPED  (max_tokens or max_model_len reached)
                  → FINISHED_ABORTED        (cancelled or disconnected)
RUNNING → PREEMPTED → back to WAITING       (memory ran out; recomputed later)

#How is a multi-user pipeline (preprocessing, postprocessing) integrated?

vLLM already does much of the per-request preprocessing and postprocessing itself, inside its API server. The rest belongs to services around it:

What makes a single engine good at serving many users at once:

• Continuous batching lets thousands of independent users share the same forward passes.
• Prefix caching across users makes a shared system prompt nearly free after the first request.
• Multi-LoRA serving lets many fine-tuned variants of one base model share a GPU, with the adapter chosen per request.
• Priorities (--scheduling-policy priority) let paying or interactive traffic go ahead of batch jobs.
• Metrics. Each server exposes Prometheus metrics at /metrics: running and waiting requests, KV cache usage, time to first token, time per output token and more.

To scale past one GPU server, a router sits in front of several replicas. Routing by prefix or by session keeps a conversation on the replica that already has its blocks cached.

A sketch of an application layer, where check_quota, retrieve and moderate stand for your own services:

async def handle(user_id: str, question: str) -> str:
    check_quota(user_id)                                   # gateway: authentication and rate limits
    context = await retrieve(question)                     # preprocessing outside vLLM (RAG)
    reply = await client.chat.completions.create(          # vLLM: template, tokenize, generate, detokenize
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "Answer using this context:\n" + context},
            {"role": "user", "content": question},
        ],
        max_tokens=512,
    )
    return moderate(reply.choices[0].message.content)     # postprocessing outside vLLM

#How can one customize the tokenizer, cache strategy, or scheduler strategy?

There are three levels: engine flags, parameters per request, and code. Most needs are covered by the first two.

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.85,   # share of VRAM for weights, activations and the KV pool
    max_model_len=8192,            # longest prompt + output allowed per request
    block_size=16,                 # tokens per KV block
    kv_cache_dtype="fp8",          # half the KV memory of FP16, on GPUs that support it
    enable_prefix_caching=True,    # reuse blocks across requests with a shared prefix
    max_num_seqs=128,              # at most 128 sequences in one step
    max_num_batched_tokens=4096,   # token budget per step: prompt chunks plus decode tokens
)

A few rules of thumb:

• The tokenizer must match training. A different vocabulary, special tokens or chat template won't crash anything. It just makes the model quietly worse, so override it only on purpose.
• Cache settings are a memory trade. More pool means more concurrent requests. FP8 KV halves the pool's cost at a small accuracy risk. A longer max_model_len needs a pool that can hold at least one request of that length.
• Scheduler settings trade throughput against latency. With priority scheduling, each request carries a priority value, and lower values run first. For policies the flags can't express, --scheduler-cls plugs in your own scheduler class, and that means working with vLLM's internals.

#Summary

vLLM serves many users at once by treating the KV cache like virtual memory and the batch like a conveyor belt. PagedAttention stores each sequence's cache in 16-token blocks found through a block table, so almost no memory is wasted and blocks can be shared. Continuous batching re-forms the batch at every step, under a token budget that mixes prompt chunks with decode tokens, so no slot sits idle and new requests start within a step. Around that core, the engine sizes one KV pool at startup, caches prefixes, preempts when memory runs out, and exposes everything through an OpenAI-compatible, streaming, asynchronous API.

#Concepts at a glance

#Test yourself

Next in the series: 06 · Batching LLM Requests: Foundations for a Queue Scheduler.