Batching LLM Requests: Foundations for a Queue Scheduler

The first five posts explained how a model runs and how an engine like vLLM serves many users at once. This post turns that into a project: a minimal LLM queue scheduler. It implements two batching strategies, naive and dynamic, and benchmarks them against vLLM's continuous batching. The project comes from Mentiforce's LLM System L3 track.

Most of the post lays the foundations the project needs: why batching works and what it costs, how requests arrive, how the strategies differ, and the five design questions of the project's pre-study phase. The last quarter is the plan, with the different ways it can be carried out. I haven't started building yet, so that part is a map of options, not a report of results.

The sixth post covers six topics:

• The project in one picture
• Why batching matters, and what it costs
• How requests arrive, and when queues explode
• The three batching strategies
• Designing the scheduler: the pre-study questions
• From plan to execution

#The project in one picture

#What will the scheduler do, and how will it be judged?

Requests arrive one by one as a random stream, wait in a queue, are grouped into batches by a scheduler, run on a model, and leave with their output and a record of how long everything took.

The acceptance criteria:

• The model can be swapped: TinyLlama-1.1B for smoke tests, and Mistral-7B or LLaMA-13B for the full runs.
• The batching strategy can be swapped: naive, dynamic or vLLM, behind one interface.
• The plots clearly show what dynamic batching improves over naive batching.

The goal is a one-line result of this form, with the numbers filled in by the benchmark: "Dynamic batching improved throughput by +X% and reduced latency by −Y% compared to naive batching on a single 3090."

#Why batching matters, and what it costs

#Why does batching give nearly free throughput?

GPUs are massively parallel, and decoding uses very little of that. Post 04 showed that each decode step reads every weight from memory to produce one token per sequence. A batch of B sequences reuses each of those reads B times. With post 04's step-time model:

t_step(B)     ≈ ( W_bytes + B × T × c_kv ) ÷ BW
throughput(B) = B ÷ t_step(B)                tokens per second, for all sequences together

For Llama-2-7B in FP16 on an A100, with 512 tokens of context per sequence:

The weights are the fixed cost of every step. While they dominate, extra sequences are almost free: 16 sequences give about 12 times the tokens per second for 1.3 times the step time. The KV cache term grows with the batch, so the gain slowly tapers off, until the batch is limited by memory (every sequence's cache must fit) or by compute.

#What is the catch?

The sequences in a batch move in lockstep. Every step produces one token for every sequence, and Hugging Face's generate() keeps going until the longest one is done:

useful decode work = Σ o_i ÷ ( B × max o_i )

outputs of 50, 120, 500 and 80 tokens:   750 ÷ (4 × 500) = 37.5%

Post 05 drew the same effect inside one static batch, together with padding. The slowest request sets the pace for everyone. The 50-token answer is ready after 50 steps but stays in the batch for 500, and the requests queued behind the batch can't start until it ends. This is head-of-line blocking.

So batching has two costs: wasted slots inside a batch, and time that requests spend waiting outside it, for the batch to fill or for the previous one to finish. The rest of this post is mostly about the second cost.

#How requests arrive, and when queues explode

#What is a Poisson arrival process, and how do you simulate one?

Real users act independently, so requests arrive at random. The standard model is a Poisson process with an average rate λ, in requests per second:

arrivals in a window of t seconds  ~  Poisson(λt)          P(k arrivals) = (λt)^k × e^(−λt) ÷ k!
gap between two arrivals           ~  Exponential(λ)       mean 1/λ,     P(gap > x) = e^(−λx)

• It is memoryless: how long you have already waited tells you nothing about when the next request will arrive.
• Bursts and lulls happen on their own. At 2 requests per second, a 2-second gap with no arrivals has a probability of e^(−4), about 1.8%, and a second with 5 or more arrivals has a probability of about 5.3%.

To simulate it, draw exponential gaps and add them up. A fixed seed gives every strategy exactly the same trace:

import asyncio
import time

import numpy as np


def poisson_arrival_times(rate: float, duration_s: float, seed: int = 0) -> list[float]:
    """Arrival times, in seconds from the start, of a Poisson process with `rate` requests per second."""
    rng = np.random.default_rng(seed)              # same seed, same trace, for every strategy
    times, t = [], rng.exponential(1.0 / rate)
    while t < duration_s:
        times.append(t)
        t += rng.exponential(1.0 / rate)
    return times


async def replay(times, make_request, scheduler):
    start = time.perf_counter()
    for t in times:
        await asyncio.sleep(max(0.0, start + t - time.perf_counter()))
        # stamp the planned arrival time, not the moment this coroutine woke up
        await scheduler.submit(make_request(arrival_time=start + t))

The picture above is one draw of that trace. Try your own: change the rate, or draw again. This chart uses its own random numbers, so its counts will not match the 33 arrivals above, but the shape holds at every rate: bursts, lulls, and gaps that average 1/λ.

The prompts themselves supply the variable lengths. Sampling prompts from a real chat dataset (vLLM's own benchmarks use ShareGPT conversations) gives a realistic mix. Drawing lengths from a chosen distribution gives more control. Either way, the same seeded trace and the same prompts go to every strategy.

#How busy can a server get before waiting explodes?

Two results from queueing theory describe every serving system.

Little's law links the arrival rate, the time in the system and the number of requests in it:

L = λ × W             L = requests in the system on average, λ = arrival rate, W = average time per request

λ = 5 requests/s, W = 4 s   →   L = 20 requests in flight

That L is the concurrency the server has to hold: room for about 20 requests in the batch and in the KV cache, or a queue that keeps growing.

Utilization is the arrival rate divided by the fastest rate the server can sustain, ρ = λ ÷ μ. In the simplest queue (one server, random arrivals and random service times, called M/M/1), the average time in the system is:

W = 1 ÷ (μ − λ)  =  (1/μ) ÷ (1 − ρ)

ρ = 0.5 → 2× the service time     ρ = 0.8 → 5×     ρ = 0.9 → 10×     ρ = 0.95 → 20×

A batching server isn't an M/M/1 queue, but the shape is universal: latency stays flat at low load and shoots up near capacity. Batching raises μ (a bigger batch serves more requests per second), which moves the knee to the right. That is why the benchmark sweeps the arrival rate (5, 20 and 50 requests per second): the interesting part of every curve is where it bends.

#The three batching strategies

#How do naive, dynamic and continuous batching differ?

The same five requests, under three strategies with room for four requests at a time. The requests arrive at steps 0, 1, 5, 6 and 7, and need 6, 3, 4, 2 and 5 decode steps:

• Naive waits for exactly four requests. Request 1 waits 6 steps for the batch to fill, then everyone runs until the longest request ends. Request 5 never starts, because three more requests would have to arrive first.
• Dynamic launches when the batch is full or the oldest request has waited 2 steps, so requests 1 and 2 start at step 2. But requests 3 to 5 arrive while that batch is running, and they wait for it to finish.
• Continuous forms the batch again at every step. Each request starts when it arrives and leaves when it is done.

Dynamic batching is a request-level policy: it decides when to launch a whole batch. Continuous batching is iteration-level: it decides at every token step, which needs an engine that owns the decode loop and the KV cache, exactly what vLLM is (post 05). That is why the project builds naive and dynamic batching itself, on top of Hugging Face's generate(), and uses vLLM as the baseline to beat.

#Designing the scheduler: the pre-study questions

Phase 0 of the project asks five questions. Unlike the questions in the earlier posts, these are about implementation. Each answer has to be precise (a data structure, an algorithm in pseudocode or a formula) and come with a reason why it is correct.

#0.1 What must a request and a response contain?

A request is read by three parts of the system: the scheduler (to form batches), the backend (to run the model) and the metrics collector (to compute latencies). So it carries the input, the generation settings, the timestamps that every metric is computed from, and its state:

import asyncio
import time
import uuid
from dataclasses import dataclass, field
from enum import Enum


class Status(Enum):
    QUEUED = "queued"
    RUNNING = "running"
    FINISHED = "finished"
    FAILED = "failed"


@dataclass
class Request:
    # input, tokenized once at submission
    prompt: str
    prompt_token_ids: list[int]
    max_new_tokens: int = 128
    temperature: float = 0.0                 # 0 means greedy: deterministic and easy to compare
    request_id: str = field(default_factory=lambda: uuid.uuid4().hex)

    # timestamps from time.perf_counter(): every latency metric comes from these four
    arrival_time: float = field(default_factory=time.perf_counter)
    scheduled_time: float | None = None      # pulled into a batch
    first_token_time: float | None = None    # first output token produced
    completion_time: float | None = None     # last output token produced

    # output and state
    output_token_ids: list[int] = field(default_factory=list)
    status: Status = Status.QUEUED
    error: str | None = None
    done: asyncio.Event = field(default_factory=asyncio.Event)   # set when finished or failed

    @property
    def prompt_len(self) -> int:
        return len(self.prompt_token_ids)

The response is what the caller receives: a frozen snapshot with the output and the metrics already computed.

@dataclass(frozen=True)
class Response:
    request_id: str
    status: Status
    output_text: str
    output_token_ids: list[int]
    prompt_len: int
    output_len: int
    queue_wait: float        # scheduled_time − arrival_time
    ttft: float              # first_token_time − arrival_time
    tpot: float | None       # (completion_time − first_token_time) ÷ (output_len − 1)
    e2e_latency: float       # completion_time − arrival_time
    error: str | None = None

PauseRequest or acknowledgmentData

Why this shape:

If the scheduler ever needs to reorder requests, two optional fields fit here later: a priority and a deadline (a TTFT target).

#0.2 How does a naive batching scheduler work?

The control flow:

1. Wait until the queue holds N requests.
2. Take exactly N, mark them running and stamp their scheduled_time.
3. Run them as one batch: pad to the longest prompt, decode until the longest output is done.
4. Fill in their tokens and timestamps, mark them finished, and set their done events.
5. Repeat. At shutdown, flush whatever is left as a smaller batch, or the last requests wait forever.

The only launch condition is len(queue) ≥ N:

loop:
    wait until len(queue) ≥ N                (at shutdown: take whatever is left)
    batch ← the N oldest requests
    for r in batch:  r.scheduled_time ← now,  r.status ← RUNNING
    run batch on the backend                 pads to the longest prompt, decodes to the longest output
    for r in batch:  record tokens and timestamps,  r.status ← FINISHED,  signal r.done

All the schedulers share one interface and one helper that runs a batch:

from abc import ABC, abstractmethod


class Backend(ABC):
    @abstractmethod
    def generate(self, batch: list[Request]) -> None:
        """Run one batch to completion, filling each request's tokens and token timestamps."""


class Scheduler(ABC):
    @abstractmethod
    async def submit(self, request: Request) -> None:
        """Called by the traffic generator when a request arrives."""

    @abstractmethod
    async def run(self, backend: Backend) -> None:
        """Form batches forever and hand them to the backend."""


async def run_batch(backend: Backend, batch: list[Request]) -> None:
    now = time.perf_counter()
    for r in batch:
        r.scheduled_time, r.status = now, Status.RUNNING
    try:
        await asyncio.to_thread(backend.generate, batch)    # the GPU call blocks: keep it off the event loop
        for r in batch:
            r.status = Status.FINISHED
    except Exception as e:                                   # a failed batch fails its requests, not the run
        for r in batch:
            r.status, r.error = Status.FAILED, repr(e)
    for r in batch:
        r.done.set()


class NaiveBatcher(Scheduler):
    def __init__(self, batch_size: int):
        self.batch_size = batch_size
        self.queue: asyncio.Queue[Request] = asyncio.Queue()

    async def submit(self, request: Request) -> None:
        await self.queue.put(request)

    async def run(self, backend: Backend) -> None:
        while True:
            batch = [await self.queue.get() for _ in range(self.batch_size)]   # wait for exactly N
            await run_batch(backend, batch)

The cost of waiting for N. Request k of a batch waits for the N − k requests that arrive after it:

E[wait of request k] = (N − k) ÷ λ             average over the batch:  E[wait] = (N − 1) ÷ (2λ)

N = 16:    λ = 1 request/s  →  7.5 s      λ = 20 requests/s  →  0.375 s      λ = 50 requests/s  →  0.15 s

On top of that come the batch's own run time and any wait for the previous batch to finish. So naive batching looks fine at high load and terrible at low load.

#0.3 How does dynamic batching decide when to flush the queue?

A dynamic batcher launches as soon as any one of these conditions holds:

1. Size: the batch has reached max_batch_size.
2. Age: the oldest waiting request has waited max_wait_ms.
3. Memory: one more request would go over the memory budget (question 0.4).

The batch size adapts on its own: it is however many requests are waiting when a trigger fires, up to the cap. There are two regimes:

GPU idle, light load:   the timer fires first      E[B] ≈ min( 1 + λ × max_wait,  max_batch_size )
GPU busy:               requests pile up during the running batch (about λ × t_batch of them),
                        and the next batch takes  min( waiting,  max_batch_size )

max_wait = 50 ms:   λ = 5/s → 1.25       λ = 50/s → 3.5       λ = 400/s → 16 (the cap)

loop:
    batch ← [ wait for the first request ]
    move every request already waiting into batch, up to max_batch_size
    deadline ← batch[0].arrival_time + max_wait
    while len(batch) < max_batch_size and memory allows one more:
        r ← wait for the next request, but not past deadline
        if none arrived in time: break
        batch.append(r)
    run batch on the backend

class DynamicBatcher(Scheduler):
    def __init__(self, max_batch_size: int = 16, max_wait_ms: float = 50):
        self.max_batch_size = max_batch_size
        self.max_wait = max_wait_ms / 1000
        self.queue: asyncio.Queue[Request] = asyncio.Queue()

    async def submit(self, request: Request) -> None:
        await self.queue.put(request)

    async def run(self, backend: Backend) -> None:
        while True:
            batch = [await self.queue.get()]                     # sleep until there is work
            while len(batch) < self.max_batch_size and not self.queue.empty():
                batch.append(self.queue.get_nowait())            # take everyone who is already waiting
            deadline = batch[0].arrival_time + self.max_wait     # the oldest request sets the clock
            while len(batch) < self.max_batch_size:              # trigger 1: size
                remaining = deadline - time.perf_counter()
                if remaining <= 0:
                    break                                        # trigger 2: age
                try:
                    batch.append(await asyncio.wait_for(self.queue.get(), remaining))
                except asyncio.TimeoutError:
                    break
            await run_batch(backend, batch)

The two knobs, and what moving them does:

Production servers expose the same two knobs. Triton Inference Server calls them max_batch_size and max_queue_delay_microseconds, and TensorFlow Serving calls them max_batch_size and batch_timeout_micros.

#0.4 How can the batch size adapt to GPU memory and queue length?

Two things grow with the batch: the KV cache, and the activations of the prefill. Because Hugging Face pads a batch to its longest prompt, every row costs as much as the longest one:

per-row KV bytes   ≈ (P_max + max_new_tokens) × c_kv           P_max = longest prompt in the batch
c_kv               = 2 × L × H_kv × d × b                      post 04: KV bytes per token

memory limit       B_mem = ⌊ (free bytes − reserve) ÷ per-row bytes ⌋
batch size         B     = min( queue length,  max_batch_size,  B_mem )

The reserve covers the prefill activations, which also grow with B × P_max. Measure it instead of guessing: run one test batch, read torch.cuda.max_memory_allocated(), and keep a margin.

For a 24 GB RTX 3090 with max_new_tokens = 128 and a 1.5 GiB reserve:

Two lessons hide in that table. With TinyLlama, the memory check never fires, so testing it needs long prompts or an artificially small budget. With Mistral-7B, a single long prompt in the batch cuts the safe batch size by more than three times.

So the heuristic adds requests oldest first and stops at the first one that would break the budget:

batch ← [],  P_max ← 0
for r in queue, oldest first, up to max_batch_size:
    P ← max(P_max, r.prompt_len)
    if (len(batch) + 1) × (P + max_new_tokens) × c_kv > budget:  stop
    batch.append(r),  P_max ← P
if batch is empty:  batch ← [oldest request]                    always make progress

on out-of-memory:        cap ← max(1, cap ÷ 2), and put the batch back in the queue     back off fast
after 20 clean batches:  cap ← min(max_batch_size, cap + 1)                             probe up slowly

import torch


def kv_budget_bytes(reserve_bytes: int) -> int:
    free, _ = torch.cuda.mem_get_info()                                    # what the driver can still hand out
    cached = torch.cuda.memory_reserved() - torch.cuda.memory_allocated()  # PyTorch's own free cache
    return free + cached - reserve_bytes


def rows_that_fit(candidates: list[Request], max_new_tokens: int, kv_per_token: int, budget: int) -> int:
    """How many of the candidates, oldest first, fit the budget once padded to the longest prompt."""
    p_max = 0
    for i, r in enumerate(candidates):
        p_max = max(p_max, r.prompt_len)
        if (i + 1) * (p_max + max_new_tokens) * kv_per_token > budget:
            return max(i, 1)                     # always make progress, even with one oversized request
    return len(candidates)

• Count PyTorch's cache as free. mem_get_info() reports what the CUDA driver can still hand out, but blocks in PyTorch's caching allocator (posts 01 and 02) count as used there, even though PyTorch can reuse them at once.
• Use a queue you can look into. asyncio.Queue can't peek at requests or put them back at the front, so the real implementation keeps a collections.deque plus an asyncio.Event to wake the batcher.
• The trade-off of grouping by length. Sorting the queue by prompt length shrinks padding, but it breaks arrival order, so it needs a cap on how long any request can be skipped.

#0.5 How are TTFT, TPOT, latency and throughput measured and logged?

Every metric comes from the four timestamps on the request:

per request
queue wait     = scheduled_time − arrival_time
TTFT           = first_token_time − arrival_time
TPOT           = (completion_time − first_token_time) ÷ (n_out − 1)          when n_out ≥ 2
E2E latency    = completion_time − arrival_time  =  TTFT + (n_out − 1) × TPOT

per run
throughput     = Σ n_out ÷ (last completion − first arrival)                 output tokens per second
request rate   = finished requests ÷ the same window                          requests per second
peak memory    = torch.cuda.max_memory_allocated(), reset before each run

What each one means to a user:

Averages hide the slow requests that users remember, so report percentiles: p50 (the median), p90 and p99.

The rules that keep the numbers honest:

• Measure TTFT from arrival, not from scheduling. Time in the queue is part of what the user waits for.
• Use one clock, time.perf_counter(), which is monotonic and high-resolution.
• GPU work is asynchronous (post 01). Call torch.cuda.synchronize() before taking a timestamp that marks GPU work as done.
• Hugging Face's streamers only support a batch size of 1. For batches, record a timestamp after every decode step with a stopping-criteria hook (below).
• For vLLM, stream from its OpenAI-compatible server and stamp each chunk on the client.
• Warm up first. The first requests include CUDA initialization and kernel compilation.
• Keep failures. Failed requests are logged with their status and error, never silently dropped.

A stopping criterion runs after every generated token, which makes it a convenient clock:

import time

import torch
from transformers import StoppingCriteria, StoppingCriteriaList


class StepClock(StoppingCriteria):
    """Records the time after every decode step. It never stops generation itself."""

    def __init__(self):
        self.stamps: list[float] = []

    def __call__(self, input_ids, scores, **kwargs):
        torch.cuda.synchronize()                 # wait until this step's GPU work is really done
        self.stamps.append(time.perf_counter())
        return torch.zeros(input_ids.shape[0], dtype=torch.bool, device=input_ids.device)


clock = StepClock()
out = model.generate(**inputs, max_new_tokens=128, do_sample=False,
                     stopping_criteria=StoppingCriteriaList([clock]))
# clock.stamps[0] is every row's first token; a row with n tokens finished at clock.stamps[n - 1]

Logging is one row per request, plus one summary per run:

import pandas as pd


def record(r: Request) -> dict:
    """One metrics row for a finished or failed request, computed from its timestamps."""
    row = {"request_id": r.request_id, "status": r.status.value, "prompt_len": r.prompt_len,
           "output_len": len(r.output_token_ids), "error": r.error}
    if r.status is Status.FINISHED:
        n = row["output_len"]
        row |= {
            "queue_wait": r.scheduled_time - r.arrival_time,
            "ttft": r.first_token_time - r.arrival_time,
            "tpot": (r.completion_time - r.first_token_time) / (n - 1) if n > 1 else None,
            "e2e": r.completion_time - r.arrival_time,
        }
    return row


def summarize(rows: list[dict], window_s: float, peak_mem_bytes: int) -> dict:
    """Per-run aggregates: throughput, failures, peak memory and latency percentiles."""
    df = pd.DataFrame(rows)
    ok = df[df.status == "finished"]
    out = {"throughput_tok_s": ok.output_len.sum() / window_s,
           "requests_per_s": len(ok) / window_s,
           "failed": int((df.status == "failed").sum()),
           "peak_mem_gb": peak_mem_bytes / 1e9}
    for metric in ("ttft", "tpot", "e2e"):
        for q in (50, 90, 99):
            out[f"{metric}_p{q}"] = ok[metric].quantile(q / 100)
    return out

Each per-request row also gets the run's settings (strategy, model, arrival rate, seed), so every plot can be made from one CSV file.

#From plan to execution

#What are the phases, and what does each one deliver?

The project page splits the work into six phases. Each has a result statement that says what "done" means:

This post is Phase 0: the five answers above are its deliverables. Phase 1 sets up the reference to compare against before any custom code exists, and Phase 2 builds the harness that every later phase reuses.

Phase 2 also sets a code-quality bar, because a framework that is slow to understand makes every later bug slower to fix:

• Clean: no redundant design or code.
• Modular: each function, class or data structure has one atomic job.
• Fast to understand: a long function becomes a short outline of 8 to 20 lines, with the details in named sub-functions.
• Reusable: one role, one unit. No two pieces of code share an overlapping responsibility.

A layout for the repository that keeps strategies and backends swappable:

llm-batching-benchmark/
├── core/
│   ├── request.py          # Request and Response dataclasses
│   ├── metrics.py          # record() and summarize(): TTFT, TPOT, throughput, memory
│   └── traffic.py          # seeded Poisson arrivals and prompt sampling
├── schedulers/
│   ├── base.py             # the Scheduler interface and run_batch()
│   ├── naive.py            # NaiveBatcher: waits for exactly N requests
│   ├── dynamic.py          # DynamicBatcher: size, age and memory triggers
│   └── vllm_adapter.py     # passes requests straight to a vLLM server
├── backends/
│   ├── simulated.py        # the step-time model, CPU only, for tests
│   ├── hf_backend.py       # Hugging Face generate() with per-step timestamps
│   └── vllm_backend.py     # OpenAI-compatible streaming client
├── benchmarks/
│   ├── run_benchmark.py    # picks scheduler, backend, model and arrival rate from a config
│   └── plot_results.py     # throughput against latency curves
├── configs/
│   ├── smoke.yaml          # TinyLlama, short runs
│   └── full.yaml           # Mistral-7B, the full sweep
└── README.md

#What are the ways to carry it out?

There is more than one way to run each part. Where to run:

Which backend:

Tools and models:

• Editor. VS Code's Remote-SSH extension and PyCharm's remote interpreters (part of its Pro tier, free for students) both run code on a remote GPU from a local editor. The code is the same either way.
• Scripts, not notebooks. Keep the library code in .py files in git, and use notebooks only to analyse results. Notebooks already run an event loop, so call await main() in a cell instead of asyncio.run(main()).
• AI tools. The project notes recommend generating code in a chat (ChatGPT) rather than with an editor agent such as Cursor that edits the repository directly, so that understanding and control stay with you.

Keeping the bill low is part of the plan:

• Develop with the simulator and TinyLlama. Switch to the larger model only for the one-time final demonstration.
• Script the whole benchmark, so the paid GPU runs unattended from start to finish.
• Write each run's results to disk as soon as it finishes, so a lost session loses one run, not the sweep.
• Keep model weights on a persistent volume, so they download once.
• Stop instances when idle. At the time of writing, consumer cards like the 3090 typically rent for well under a dollar an hour, but prices change, so check before choosing.

#How should the benchmark be run so the comparison is fair?

The differences in the results should come from the scheduler and nothing else:

• The same arrival trace (the same seed) and the same prompts for every strategy.
• The same output lengths. Use greedy decoding and either ignore the end-of-sequence token or give each request a target length that every backend enforces. Otherwise one strategy may simply get shorter answers.
• Warm-up, duration and repeats. Warm up first, run each point for at least 60 seconds of traffic, and repeat it at least three times.
• A sweep past the knee. Use several arrival rates (5, 20 and 50 requests per second), including at least one that overloads the naive scheduler.
• Comparable memory numbers. vLLM reserves its whole pool up front, so its memory use sits flat at about gpu_memory_utilization × VRAM. Compare its KV cache usage (from /metrics) with Hugging Face's peak allocation, and say which number is which.
• The full setup in the report: GPU, model, dtype, library versions and the scheduler's knob values.

Phase 1 starts with a client that measures a vLLM server:

vllm serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --gpu-memory-utilization 0.9

import time

from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"


async def timed_request(prompt: str, max_tokens: int = 128) -> dict:
    t0 = time.perf_counter()
    stamps, usage = [], None
    stream = await client.completions.create(
        model=MODEL, prompt=prompt, max_tokens=max_tokens, temperature=0.0,
        stream=True, stream_options={"include_usage": True},
    )
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].text:
            stamps.append(time.perf_counter())   # one stamp per streamed piece of text
        if chunk.usage:
            usage = chunk.usage                  # the final chunk reports exact token counts
    if not stamps:
        return {"status": "failed"}
    n = usage.completion_tokens if usage else len(stamps)
    return {"status": "finished", "ttft": stamps[0] - t0, "e2e": stamps[-1] - t0,
            "tpot": (stamps[-1] - stamps[0]) / (n - 1) if n > 1 else None, "output_len": n}

A streamed chunk can carry more than one token, which is why TPOT divides by the exact token count from the usage report, not by the number of chunks.

#What results should we expect?

Before measuring anything, the theory above predicts the shape of the results. These are hypotheses for the benchmark to confirm or refute:

• Low load: naive batching has a very long TTFT, because batches take a long time to fill. Dynamic batching and vLLM start requests almost at once.
• High load: naive and dynamic batching both form full batches, so their throughput is similar, and both suffer from head-of-line blocking.
• Everywhere: vLLM should lead on throughput and tail latency, thanks to continuous batching and paging, with its knee furthest to the right.
• The headline number depends on the load chosen. The fair claim is a curve, and the one-line result should name the arrival rate it was measured at.

#Summary

Batching turns a memory-bound decode into nearly free throughput, because one read of the weights serves every sequence in the batch. Its costs are lockstep (a batch runs as long as its longest request) and waiting (for a batch to fill or to finish). Requests arrive as a Poisson stream, and queueing theory says latency explodes near capacity. A naive batcher waits for exactly N requests, a dynamic batcher launches on size, age or memory, and a continuous batcher (vLLM) re-forms the batch at every step. The project's design rests on a request object that carries its own timestamps, two small asyncio schedulers, a padding-aware memory check, and metrics computed from those timestamps, measured fairly across a sweep of arrival rates.

#Concepts at a glance

#Test yourself

The next step is Phase 1: getting the vLLM baseline running and measured.