Neural Network Fundamentals

The first post stayed on the Python side of things. This one moves to the neural network itself: what a network is made of, how its parameters live in memory, how a forward pass runs on the GPU, and why inference needs so much less memory than training.

The second post covers four topics:

• What are the basic components of a neural network?
• How does a model run during inference?
• Do the optimizer and gradients matter for inference?
• Memory usage and VRAM allocation during inference

#What are the basic components of a neural network?

#What is a layer? What is a parameter?

#What are the components of a neural network model?

A neural network is a highly flexible function approximator made of many interconnected computational elements. A neuron (or node) is the fundamental processing unit. Each neuron receives one or more inputs, computes a weighted sum, and applies an activation function to the result.

Neurons are organised into layers:

• The input layer receives the raw data. It does no computation other than passing the input features through.
• Hidden layers perform complex feature extraction and transformation, through successive weighted sums and activation functions.
• The output layer produces the final predictions or classifications.

• Without non-linearity, a deep network would collapse into a single linear transformation, which severely limits its expressive power.
• The goal of training is to minimise the loss over the training data, and the loss guides the parameter updates.
• Optimizers typically use gradient-based methods, such as Stochastic Gradient Descent (SGD) and its variants (Adam, RMSprop), to update the parameters step by step in the direction that reduces the loss.

You can see the parameters of a tiny model directly in PyTorch:

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(4, 8),  # input to hidden: an 8 x 4 weight matrix plus 8 biases
    nn.ReLU(),        # an activation function: it has no learnable parameters
    nn.Linear(8, 2),  # hidden to output: a 2 x 8 weight matrix plus 2 biases
)

for name, p in model.named_parameters():
    print(name, tuple(p.shape))  # each one is a trainable parameter tensor

print(sum(p.numel() for p in model.parameters()))  # 4*8 + 8 + 8*2 + 2 = 58 parameters

#Which ones are trainable parameters, and which are constants?

In short, trainable parameters are the adaptive elements that encode what the network has learned. Constants (hyperparameters) are the design choices around them.

#Which are used during the forward pass, and what extra is used during training?

PauseDataMetadata

The training phase extends the forward pass with ways to learn from errors and adjust the learnable parameters step by step.

• Loss function. A scalar-valued function that measures the gap between the prediction and the true label. It gives optimization something measurable to aim at, and a lower loss means a better model.
  • Examples: Mean Squared Error (MSE) for regression, and Cross-Entropy loss for classification.
• Gradients. Vectors of partial derivatives of the loss with respect to each learnable parameter. The optimizer uses them to decide how to adjust the parameters so the loss goes down.
  • They are computed efficiently by the backpropagation algorithm, which uses the chain rule of calculus.
  • The core idea is to propagate the error backwards through the network, layer by layer, calculating the gradient for each parameter.
• Optimizer. Uses the computed gradients to update the network's learnable parameters step by step and reduce the loss.
• Learning rate. A crucial hyperparameter that controls the step size the optimizer takes when adjusting parameters, in other words how aggressively the model learns.
  • A learning rate that is too high can cause instability and divergence. One that is too low can mean slow convergence, or getting stuck in a local minimum.

#How are parameters stored? How does PyTorch manage them?

#How are parameter tensors allocated in memory?

• Parameters are n-dimensional arrays, managed by the deep learning framework's runtime and hardware interfaces. They hold numerical data and are tied to a specific device and data type.
• torch.Tensor is the fundamental data structure, a multi-dimensional matrix. It acts as a handle to a block of memory that lives either on the CPU (host memory) or on a GPU (device memory).

• Device transfer. Moving a tensor from CPU to GPU (with tensor.to('cuda') or model.cuda()) copies its data from host RAM to device VRAM. This is a crucial step for using parallel computation, and the transfer bandwidth can be a significant bottleneck.
• Memory layout and contiguity. Tensors are often stored in one contiguous block to optimise access patterns and efficiency. PyTorch uses strides to handle non-contiguous views without copying data, and an explicit .contiguous() call forces contiguity when needed, possibly at the cost of a copy.
• Data types and precision. The memory allocated is directly proportional to the number of elements and the size of the data type. Lower-precision types reduce the memory footprint and can speed up computation on compatible hardware.

• Parameter management in nn.Module. When you define a network with torch.nn.Module, parameters are registered with self.register_parameter() (or implicitly through nn.Parameter). That lets the module track them, move them between devices, and include them in the optimizer's parameter list for gradient updates.

#Do all parameters have to be moved to the GPU? When does this happen?

• PyTorch operations are device-agnostic in their definition: the same operation can run on the CPU or the GPU. But all the tensors taking part must be on the same device. You can't directly combine a CPU tensor with a GPU tensor.
• Not all parameters must be on the GPU, but for a computation to use GPU acceleration, the relevant parameters and input data must sit on the same GPU as the operation.
• Parameters move to the GPU when the user asks for it explicitly, or when a framework (for example for distributed training) arranges the placement. This usually happens just before or at the point of computation.

model = model.to("cuda")  # moves every parameter and buffer of the module
x = x.to("cuda")          # the inputs must be on the same device as the model
y = model(x)              # now the computation can run on the GPU

• Explicit control. Individual tensors move with .to(device) or its alias .cuda(). For a torch.nn.Module, module.to(device) moves all the module's parameters and buffers (such as batch normalization's running statistics).
• Placement in more advanced setups. Not every parameter has to sit on the GPU all the time. The multi-GPU strategies from post 01 (data, tensor and pipeline parallelism, and sharding with ZeRO or FSDP) spread parameters across GPUs. Offloading goes one step further: parameters that aren't needed yet wait in CPU memory and are copied to the GPU just before their layer runs. Hugging Face's device_map="auto" does this when a model doesn't fit, trading speed for memory.

#Can multiple models share parameters?

• Yes. Parameter sharing (also called weight sharing) brings efficiency, regularisation and shared representations across different parts of a network, or across separate models.
• It is an architectural pattern in which a single set of trainable parameters (weights and biases) is used by several computational units or layers in a network, or even across model instances.
• The principle behind it is an inductive bias. If certain transformations are expected to be useful everywhere, for example at different locations in an image or across related tasks, sharing parameters lets those components learn one robust, unified representation.
• In PyTorch, sharing works through Python's object references and the module system: several nn.Module instances or sub-modules reference the same torch.nn.Parameter objects. If a sub-module is passed by reference to several parents, they all operate on exactly the same tensors. During backpropagation, the gradients for shared parameters accumulate correctly.

embed = nn.Embedding(1000, 64)           # a 1000 x 64 embedding matrix
proj = nn.Linear(64, 1000, bias=False)   # its weight is also 1000 x 64
proj.weight = embed.weight               # both now share one Parameter (weight tying)

Common patterns:

#How does a model run during inference?

#How is the forward pass executed on the GPU?

#How does a Linear layer → matrix multiplication run?

A Linear layer transforms an input matrix (or a batch of vectors) X into an output Y, using a weight matrix W and an optional bias vector B: Y = X · Wᵀ + B.

The GPU execution paradigm is massive parallelism. GPUs are designed for Single Instruction, Multiple Threads (SIMT) execution, which suits matrix multiplication perfectly, because each output element can be computed independently. Computation is organised in a hierarchy:

For the general matrix multiplication C = A × B, where A is M × K and B is K × N, the output C is M × N. Each element C[i][j] is the dot product of row i of A and column j of B.

• A common strategy maps portions of the output matrix C to thread blocks. A block might be responsible for computing one tile (a sub-matrix) of C, and within the block each thread computes specific elements or parts of elements.
• To maximise efficiency and minimise slow global memory access, highly optimised kernels use tiling with shared memory:
  1. Large matrices A and B are divided into smaller tiles that fit into the GPU's fast on-chip shared memory.
  2. The threads in a block load the matching tiles of A and B into shared memory together. This happens once per tile, which spreads the cost of the global memory access over many uses.
  3. Once the tiles are in shared memory, the threads perform many dot products (or partial dot products) on the fast shared data. This gives a high ratio of computation to memory access.
• This dramatically reduces memory latency, lets data be reused, and keeps the GPU's arithmetic logic units (ALUs) busy, reaching near peak performance. Individual threads also use fast registers for intermediate calculations, and compilers and hardware pipeline the instructions aggressively to keep execution continuous.

Deep learning frameworks hide these low-level details and rely on highly optimised libraries to run matrix multiplications:

When a linear layer is defined and run on a GPU, PyTorch, TensorFlow and JAX automatically dispatch the matrix multiplication to the most suitable cuBLAS or cuDNN routine, often using Tensor Cores when they are available.

#Is each operation a separate kernel call?

• Not necessarily for every numerical computation. Conceptually, distinct operations can map to individual GPU kernel calls, but modern frameworks use optimisation techniques to cut overhead and maximise parallelism.
• Launching a GPU kernel from the CPU has a cost. It includes CPU-GPU communication (copying parameters, setting up launch configurations), kernel enqueueing, and synchronisation if later operations depend on this kernel finishing.
• For a sequence of many small operations, the launch overhead adds up and can become a significant bottleneck, especially for memory-bound work.
• There are two ways to cut that overhead:
  • Kernel fusion merges consecutive operations into one kernel (next question).
  • CUDA graphs record a whole sequence of kernel launches once, then replay it with a single launch. The kernels stay separate, but the CPU cost of launching them almost disappears. vLLM captures CUDA graphs for its decode steps, where every kernel is tiny and launch overhead would otherwise dominate.

#Is there kernel fusion? Can multiple layers be merged?

• Yes. Kernel fusion lets several elementary operations (layers, or parts of layers) merge into a single GPU kernel. It combines a sequence of operations that would normally need separate kernel launches into one larger kernel.
• Standard execution model. Without fusion, each operation (matrix multiplication, ReLU, bias addition) goes through: launch a new kernel on the GPU → load the input data from global memory → compute → write the intermediate result back to global memory.
• Fused execution model. With fusion, a compiler finds chances to chain these operations directly in the GPU's faster on-chip memory (registers, shared memory) for the intermediate results, avoiding costly round trips to global memory.

#How does memory layout affect performance?

#Is PyTorch column-major or row-major?

• Row-major (C-style): the elements of the last dimension are stored next to each other in memory. When you walk along a row, the elements are adjacent.
• Column-major (Fortran-style): the elements of the first dimension are stored next to each other. When you walk down a column, the elements are adjacent.
• PyTorch tensors are row-major by default, like NumPy arrays.
  • PyTorch uses strides to describe how to move through a tensor's memory. A stride is the jump in memory (in number of elements) needed to get to the next element along one dimension.
  • A tensor is contiguous if its elements are stored in the same order as a row-major tensor of the same shape. Operations like transpose() or permute() often create non-contiguous views by changing the strides without rearranging the underlying data.
    • Check contiguity with tensor.is_contiguous().
    • Force contiguity with tensor.contiguous(). Some operations (convolutions, view reshaping) perform best, or only work, on contiguous data. This creates a new tensor with the data physically rearranged in memory.

import torch

x = torch.arange(1, 7).reshape(2, 3)   # [[1, 2, 3], [4, 5, 6]]
print(x.stride(), x.is_contiguous())    # (3, 1) True

t = x.t()                               # a transposed view: no data is copied
print(t.stride(), t.is_contiguous())    # (1, 3) False

c = t.contiguous()                      # the data is now physically rearranged (a copy)
print(c.stride(), c.is_contiguous())    # (2, 1) True

Performance implications for neural networks. Memory layout matters because of cache locality and vectorisation efficiency on modern hardware such as GPUs:

• Reading contiguous regions lets the CPU or GPU use its cache hierarchy well. When data is read linearly, prefetching can bring the next data into the cache before it is asked for, which reduces latency.
• Modern processors and GPUs work on many data elements at once with SIMD instructions. Contiguous access patterns are crucial for these vector units to load and process data efficiently, without the penalties of scattered memory access.

#Why perform layer fusion?

The previous answers showed what fusion does. Here are the reasons for it, side by side:

• Example: Convolution + ReLU fusion. Instead of running a convolution and then a separate ReLU, a fused kernel computes both in sequence for each output element of the convolution, and writes the ReLU output directly to global memory.

#How are activations stored in intermediate buffers?

• Activations are the outputs of the non-linear activation functions applied after linear transformations (convolutions, matrix multiplications) inside a layer. They are the transformed feature maps passed as input to the next layer.
• Intermediate buffers are temporary memory regions that hold activations between successive operations or layers. On GPUs they typically sit in global device memory, but the frequently used parts are often staged into faster on-chip memory such as shared memory or registers.
• Good storage uses the GPU's multi-level memory hierarchy, so data is as close as possible to the processing units (SMs and cores) when needed, which minimises expensive global memory access:

Two common layouts for 4D image tensors:

• NCHW (channels-first): Batch × Channels × Height × Width. Width varies fastest in memory, so each channel's H × W plane is stored contiguously.
• NHWC (channels-last): the channels are the innermost dimension (Activations[N, H, W, C]). This gives better spatial locality for pixel-wise operations (element-wise additions, non-linearities), because all the channel values for one (H, W) position sit together.

Advanced buffer management techniques:

• Tiling and blocking. The same tiling as in the matrix multiplication above: activations are processed in tiles that fit in shared memory.
• Buffer reuse and in-place operations. Frameworks reuse existing buffers for intermediate results where possible, instead of allocating new ones. In-place operations (such as relu_(x) in PyTorch) modify the buffer directly, which reduces memory traffic and allocations.
• Quantization. Storing activations at lower precision (FP16, INT8) cuts the memory footprint a lot, lets more data fit in faster caches and lowers bandwidth needs. It is a common inference optimisation on specialised hardware.
• Dynamic paging and virtual memory. For extremely large models, researchers are exploring techniques where only the active parts of the model (including activations) stay in GPU memory, like CPU virtual memory, to handle models larger than the GPU's VRAM.

#Do optimizer and gradients matter for inference?

#Is gradient needed during inference?

#Does inference generate intermediate results? Where are they stored?

• Yes. The storage location of these intermediate results is a critical factor for inference performance, and it is mainly determined by the device doing the computation.
• What they are: mostly the activations (feature maps) produced by each layer as data moves through the network. A layer's output transforms the input it received and becomes the next layer's input.
• Why they exist: intermediate activations are indispensable for the layer-by-layer computation of the forward pass. Each layer applies its transformations (convolutions, matrix multiplications, non-linear activations) to its input and produces a representation with progressively more abstract features. This continues until the final layer computes the network's prediction or decision.

#Why does .eval() change model behavior (dropout, normalization)?

• .eval() turns off the stochastic parts of the model (such as Dropout) and makes statistics-dependent parts (such as Batch Normalization) use stable, learned population statistics instead of batch-specific ones. This is critical for consistent, reliable predictions.
• It switches the model into evaluation mode, which differs from training mode. The difference matters for layers that behave differently in training and inference.

model.eval()               # dropout off; batch norm uses its running statistics
with torch.no_grad():      # also skip building the autograd graph (more on this below)
    out = model(x)

#Does the optimizer consume resources?

#How does AdamW keep momentum for each parameter?

• AdamW doesn't keep a single "momentum" scalar. For each parameter it keeps a first moment vector (an average of past gradients), which gives a momentum-like effect, and a second moment vector (an average of squared gradients), which provides adaptive learning-rate scaling.
• Both are maintained independently for every parameter, which allows nuanced and efficient optimisation.

opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
# After the first opt.step(), opt.state holds, for every parameter, two extra tensors of the
# same shape: "exp_avg" (the first moment) and "exp_avg_sq" (the second moment).

#How does the optimizer "scan" through all parameters?

• It is a highly parallel operation, usually run efficiently on specialised hardware such as GPUs.
• During the backward pass (backpropagation), gradients are computed for all parameters. Each parameter tensor has a matching gradient tensor with the same shape.
  • The framework builds a computational graph that records all operations of the forward pass. During backpropagation the graph is traversed in reverse, applying the chain rule to compute all the gradients efficiently.
• The optimizer's main job is to take these gradients and apply a predefined update rule to each parameter. The rule changes the parameter's value based on its gradient and possibly other state (such as momentum).
• The efficiency of this "scan" comes from vectorisation and parallelisation. Instead of separate CPU loops, operations run on entire tensors at once.
  • Frameworks translate these mathematical operations into highly optimised CUDA kernels (on NVIDIA GPUs) or similar low-level routines. One kernel can update millions of parameters at the same time. PyTorch's optimizers also offer foreach and fused implementations that group many tensors into few kernels.
  • Conceptually an optimizer iterates over the parameters, but in practice it dispatches tensor-level operations. A simple SGD update, param.data -= learning_rate * param.grad, runs as a single parallel element-wise subtraction and scaling on the GPU.

#Why is optimizer memory usage 2–4× the parameter size?

• Mainly because optimizers must keep extra state variables for every parameter, together with numerical-stability needs, especially in mixed-precision training.
  • Modern optimizers such as Adam, AdamW and RMSprop don't store only the parameter values. They keep internal state for each parameter to make updates more effective and stable.
  • The bytes are easiest to count in mixed-precision training, a common practice that reduces memory use and speeds up computation, especially on modern GPUs (for example NVIDIA Tensor Cores).
    • The model's parameters may be stored in FP16 (half precision, 2 bytes) to save memory and use specialised hardware, while the optimizer's state variables (such as m and v in Adam) are typically kept in FP32 (single precision, 4 bytes).
    • In many mixed-precision setups (such as NVIDIA's Apex AMP or PyTorch's Automatic Mixed Precision), an extra FP32 master copy of the weights may also be kept. Gradients are typically computed in FP16, scaled, then converted to FP32 to update these master weights, which are then cast back to FP16 for the model.

Counting the bytes for one parameter with AdamW:

• The optimizer state alone (m and v, 8 B) is 2× the FP32 weight size. Adding the gradients makes the total 4×. That is where "2–4×" comes from.
• For a 7-billion-parameter model that is 16 B × 7 × 10⁹ = 112 GB for training, before any activations, against 14 GB to serve it in FP16.

#Memory usage and VRAM allocation during inference

#How much memory is taken by parameters, inputs, and computation graphs?

#Model weights take fixed space, but what about input tensors?

• Input tensors are dynamic and add a lot to VRAM use during inference, mainly through batch size and input dimensions. The memory footprint changes with each request, depending on the characteristics of the input.

Memory = batch size × input elements per sample × bytes per element

• During inference, the input tensor memory is allocated on the GPU. It is often smaller than the model weights, but it becomes critical for:
  • High-throughput scenarios that use large batch sizes.
  • Models that process very high-resolution images or long sequences.
  • Edge or embedded devices with limited VRAM.

#Do intermediate activations take VRAM? Are there release strategies?

• Yes. Intermediate activations are the outputs of hidden layers or operations inside a network, and they serve as inputs for later layers in the computational graph. There are several ways to release or reduce them:

#Why does VRAM fill up when batch size increases?

• The weights and biases of the network stay constant whatever the batch size.
• The memory for the input data scales linearly with batch size. For a batch of B, it needs B × S_input.
• Intermediate activations: each layer produces activations for every item in the batch. If one layer's activation for one sample has size S_act, a batch of B needs B × S_act to hold them at the same time.

Putting these together:

Total VRAM ≈ parameters + B × (input per sample) + B × (activations per sample, summed over all layers) + overhead

Only the parameters stay fixed. Everything else grows with the batch size B, so increasing B steadily raises VRAM consumption.

#Differences in memory usage between inference and training

#Does inference skip gradient storage?

• Yes, because inference runs only a forward pass, to produce predictions from the weights.
• Gradient computation is an integral part of backpropagation, which works out how much each parameter contributed to the prediction error.
• A pre-trained model is simply applied, with no parameter updates. So the intermediate values that backpropagation would need aren't kept beyond their immediate use in the forward pass.
• Without gradient storage, the VRAM footprint shrinks, and much larger models can be deployed for inference on hardware with far less memory than their training needed.

#Do dropout and batchnorm affect memory usage?

• Dropout. During training, dropout randomly sets a fraction of the inputs to zero at each update, to prevent co-adaptation of features. To keep backpropagation consistent, the specific pattern of dropped neurons (the dropout mask) must be stored. During inference there is no random dropping, so no dropout masks are generated or stored.
• Batch normalization. During training, BatchNorm normalises each batch with that batch's own statistics (which must be kept for the backward pass), and it updates running estimates of the mean and variance along the way. Those running statistics are saved as part of the model's state. During inference, BatchNorm just uses the stored running statistics, so there are no per-batch statistics to compute or keep.

#Training needs optimizer states. Does inference?

• No. Inference is a forward pass through fixed weights, so the machinery for updating them (gradients and optimizer states) is never allocated. The table in the optimizer section above shows the saving: 16 bytes per parameter for AdamW training, against 2 for FP16 inference.
• In specialised settings where models keep adapting (certain online learning systems, meta-learning), a model may do an "inference" step immediately followed by a "micro-training" step. Then a minimal set of optimizer states may be kept or re-initialised. This departs from the conventional meaning of static inference.
• Optimizer states are different from recurrent states or attention key/value caches in sequential models. Those belong to the model's architectural design, keeping context across time steps or tokens during inference, and have nothing to do with optimization. For LLMs the KV cache is the big one, and post 04 is about it.

#Causes of VRAM OOM (out-of-memory) and prediction

#Is there a way to estimate the VRAM required for a given model and batch size?

During inference, VRAM is mostly used by three categories of data, plus a baseline overhead:

There are two ways to estimate it:

• Analytical approach: work out each component from the model architecture.
  • Calculate the total number of parameters and the chosen data type. Calculate the input and output memory from the batch size and tensor shapes.
  • Trace the computation graph by hand to find the largest intermediate tensors, or use introspection tools that report layer-wise output shapes.
• Empirical approach: load the model and run actual inference on the target hardware to measure memory use.
  • Load the model onto the GPU and measure the base memory. Run a forward pass with a small batch size and record the maximum VRAM allocated.
  • Extrapolate the parameters and activations. A simple linear scaling of the difference is often used, but a more accurate way is to test several batch sizes and watch how memory scales.

import torch

torch.cuda.reset_peak_memory_stats()
model = model.to("cuda").eval()
base = torch.cuda.memory_allocated()               # the weights are loaded

x = torch.randn(8, 512, device="cuda")             # a batch of 8 (adjust the shape to your model)
with torch.no_grad():
    model(x)                                       # one forward pass

peak = torch.cuda.max_memory_allocated()           # the highest allocation seen so far
print(f"weights: {base / 1e9:.2f} GB, peak: {peak / 1e9:.2f} GB")

#Can torch.compile or torch.no_grad save memory?

• PyTorch's automatic differentiation engine, Autograd, builds a computational graph dynamically during the forward pass. That graph retains intermediate activations and the metadata needed to compute gradients in the backward pass.
• Inside a torch.no_grad() block, graph construction and gradient tracking are switched off. So intermediate tensors that would otherwise be kept for backpropagation are either never allocated for gradient purposes or freed at once, and no memory is allocated for gradient storage.

with torch.no_grad():                 # or torch.inference_mode(): no autograd graph is built
    y = model(x)

fast_model = torch.compile(model)     # may fuse kernels: faster, and sometimes smaller in memory

#Can temporary intermediate tensors be freed early?

• It depends on whether the intermediate tensors are still needed for later computation, particularly for gradient calculation during training.
• During inference: with torch.no_grad() (above), each intermediate tensor is freed as soon as the next layer has used it.
• During training: activation checkpointing frees most activations during the forward pass and recomputes them during the backward pass, trading computation for memory. Splitting an n-layer network into segments of about √n layers and keeping only the segment boundaries cuts activation memory from O(n) to about O(√n), for roughly one extra forward pass.

from torch.utils.checkpoint import checkpoint


def forward(self, x):
    for block in self.blocks:
        x = checkpoint(block, x, use_reentrant=False)   # keep only x, recompute inside block on backward
    return x

#How is VRAM managed for multiple models and users on the same GPU?

#How does PyTorch know which tensors are still alive?

• Through the interplay of Python's reference counting and PyTorch's dynamic computational graph (the autograd engine).
• For tensors with requires_grad=True, a dynamic computation graph is built on the fly for the backward pass. A tensor that tracks gradients has a .grad_fn attribute pointing to the function that created it. Intermediate tensors are implicitly kept alive until their role in gradient computation is done.
• A tensor's memory is freed when its Python object's reference count drops to zero. The tensor is then destroyed, and its destructor releases the underlying C++ and CUDA memory.

Practical implications and controls:

• Disable gradient tracking with torch.no_grad() or torch.inference_mode(), as above, so that the graph doesn't keep activations alive.
• Detach tensors. tensor.detach() creates a new tensor that shares the same data but is completely removed from the current computation graph. It is useful to stop gradients flowing back through part of the graph, or to save memory on intermediate results you don't need for backpropagation.
• Delete explicitly. Setting a variable to None, or using del on a tensor, reduces its reference count, so Python can reclaim the memory earlier, provided nothing else references it.

#What causes VRAM fragmentation?

• VRAM fragmentation is the state in which free GPU memory is scattered in small, non-contiguous pieces. A new tensor that needs one large contiguous block can't be allocated, even if the total free memory is enough.
• Unlike CPU RAM, GPU memory often prefers contiguous allocation for performance. When a program requests a block, the allocator tries to find a single contiguous region. If only small separated regions are free, a larger allocation fails.

The main causes:

• Dynamic tensor lifecycles. Tensors are allocated and freed at different times, especially during training or with variable-sized inputs.
• Varying tensor sizes. Allocating and freeing blocks of different sizes over time naturally leaves uneven free space.
• Concurrent operations and multi-tenancy. Sharing a GPU between several models or processes quickly produces an interleaved pattern of allocated and freed blocks, which speeds up fragmentation.
• No memory compaction. GPU allocators typically don't compact memory in real time because of the performance cost, so fragmentation persists.

How it shows up:

• The most common symptom is an OOM error even when nvidia-smi (or an equivalent) shows plenty of "free" VRAM. That free VRAM just isn't contiguous enough for the next large allocation.
• Allocators may also spend more time searching for suitable blocks, which can hurt performance, though OOM is the more critical problem.

#How do you manually free memory (del, torch.cuda.empty_cache()) for actual release?

It takes two steps, because there are two layers. Python's reference counting decides when a tensor dies, and PyTorch's caching allocator (post 01) decides when its memory goes back to the driver.

1. Drop every reference to the tensors you no longer need, with del or by letting them go out of scope. The block returns to PyTorch's cache, ready for the next tensor in this process.
2. Call torch.cuda.empty_cache() to hand the unused cached blocks back to the CUDA driver, so that other processes can use them.

del big_tensor            # drop the last reference: the block returns to PyTorch's cache
torch.cuda.empty_cache()  # now hand the cached, unused blocks back to the CUDA driver

When several models or users share one GPU, two more facts matter:

• A memory cap per process. torch.cuda.set_per_process_memory_fraction(0.5) makes this process's allocator raise an out-of-memory error rather than take more than half of the GPU.
• The CUDA context is per process. Every process that touches CUDA holds its own context, a few hundred megabytes that empty_cache() can't release. Only ending the process gives it back.

#Summary

A network is layers of weights and biases with non-linear activations between them, and training adds a loss, gradients and an optimizer on top of the forward pass. Parameters are tensors with a shape, strides, a dtype and a device, and every tensor in an operation must be on the same device. On the GPU, a Linear layer is a tiled matrix multiplication, and fusion and CUDA graphs cut the cost of many small kernels. Training with AdamW needs about 16 bytes per parameter, while FP16 inference needs 2, and torch.no_grad() drops the autograd graph. VRAM holds the weights, plus inputs and activations that grow with the batch, plus overhead, and fragmentation can cause an out-of-memory error even when the total free memory looks sufficient.

#Concepts at a glance

#Test yourself

Next in the series: 03 · Transformer Architecture Principles.