Python Internals for LLM Engineers

#About this series

This series started as part of my learning on LLM internals. In mid 2025 I started my journey into how LLMs work inside, and that is when I came across Mentiforce through a Reddit post. Each post walks through one stage of what I learnt, from Python internals all the way to the vLLM framework and inference engine design, and the last one lays the foundations of the project I'm building next: a small scheduler for LLM requests.

The first post covers four topics:

• What is Python, and why do inference frameworks choose it?
• Python and the OS: process, memory and syscalls
• Python and the CPU: where does computation happen?
• How Python calls the GPU

#What is Python? Why do inference frameworks choose it?

#How is a Python program executed?

#How is the python3 command found and executed?

• The OS searches for an executable file named python3 along the directories in the PATH environment variable.
• After that, since Python is an interpreted language, the interpreter takes over. It reads the script, compiles it to bytecode and executes it, in a series of well-defined steps.

#What does Python do when parsing app.py? What are the stages (compile, eval)?

There are four stages between your source file and a running program:

1. Lexing. The lexer (or scanner) reads the source code character by character and groups the characters into meaningful sequences called tokens. A token is a distinct unit such as a keyword, identifier, operator or literal.
   • This converts a raw string of characters into a structured sequence that the parser can process.
2. Parsing. The stream of tokens is checked against Python's grammar rules, to make sure the code is syntactically correct, and an Abstract Syntax Tree (AST) is built. The AST represents the hierarchical structure of the program and the relationships between its parts.
   • Each node in the tree is a construct in the source code, such as an expression, statement or declaration.
   • It is "abstract" because it leaves out concrete syntax such as parentheses and semicolons, and keeps only the structure that matters for interpretation.
   • It captures the relationships and dependencies between different parts of the code, which makes it a good intermediate representation for further processing.
3. Compilation. The AST is converted into bytecode, a sequence of instructions for the Python Virtual Machine (PVM). The bytecode is wrapped in code objects, which can be cached in .pyc files so later runs load faster.
   • Bytecode is a sequence of low-level, platform-independent instructions designed to be executed by the PVM.
   • It is an intermediate representation: more abstract than machine code, but more concrete than the source code or the AST.
   • It provides a layer of abstraction that makes Python's "write once, run anywhere" possible.
4. Execution. The PVM takes the bytecode instructions from the code objects and executes them.

PauseData

You can see the first three stages yourself. This sketch uses only the standard library:

import ast
import dis
import io
import tokenize

source = "total = price * qty"

# Stage 1, lexing: the source text becomes a stream of tokens
for tok in tokenize.generate_tokens(io.StringIO(source).readline):
    print(tokenize.tok_name[tok.type], repr(tok.string))

# Stage 2, parsing: the tokens become an abstract syntax tree
tree = ast.parse(source)
print(ast.dump(tree, indent=2))  # indent=2 pretty-prints the tree

# Stage 3, compilation: the tree becomes a code object holding bytecode
code = compile(source, "app.py", "exec")  # "exec" means: compile a whole module
dis.dis(code)                              # disassemble: print the bytecode instructions

The built-in eval() function exposes the same pipeline at runtime. It takes a Python expression as a string, parses it, compiles it into a code object and has the PVM execute it:

print(eval("2 + 3 * 4"))  # parsed, compiled and executed at runtime: prints 14

#What is Python bytecode? Is there an intermediate language?

• Bytecode is a low-level, platform-independent representation of Python source code. The interpreter first compiles the source into bytecode, which the PVM then executes.
• Bytecode is an intermediate language. It is the bridge between high-level Python source code and the machine-specific instructions that a CPU can execute.
• It is a crucial step for Python's portability: the same bytecode runs on different operating systems and architectures, as long as a compatible PVM is available.
• When the PVM "executes" bytecode, it doesn't typically translate each instruction into machine instructions on the fly (that would be just-in-time compilation). Instead, the PVM's pre-compiled native code performs the operation that each bytecode instruction describes.
• Since Python 3.11 the interpreter is also specializing. After an instruction has run a few times, it is swapped for a faster version for the types it actually sees: a BINARY_OP that keeps adding two integers becomes an integer-only add. dis.dis(code, adaptive=True) shows the specialized instructions. Python 3.13 added an experimental JIT compiler as well, off by default.

#What is Python's relationship with C?

#What is CPython? Why is it written in C?

• CPython is the reference implementation of the Python language. When people say "Python", they typically mean CPython. It is written primarily in C and Python.
• The core interpreter, parser, bytecode interpreter and runtime system are implemented in C.
• The flow is CPython → bytecode → a C-implemented VM, which manages Python's objects, memory and calls to the OS.

Understanding CPython's C foundation also explains the Global Interpreter Lock (GIL), which protects access to Python objects and stops multiple threads from executing bytecode at once.

#When we use torch or numpy, who is actually running the code?

• Commands like np.array(...) or torch.matmul(...) are a high-level interface and orchestrator. They handle object creation, memory management (at the Python level) and the dispatching of calls to the routines that do the real computation.
• NumPy's backends link to BLAS (Basic Linear Algebra Subprograms) and LAPACK (Linear Algebra Package) implementations such as OpenBLAS, Intel MKL (Math Kernel Library) or ATLAS. These are highly optimised C, C++ and Fortran libraries that use instruction sets like AVX and AVX-512.
• PyTorch's CPU backend uses Intel MKL or OpenBLAS. On the GPU, NVIDIA's CUDA toolkit libraries such as cuBLAS and cuDNN come into the picture.
• CPython's ability to integrate with highly efficient C and C++ libraries is one of its most powerful advantages for high-performance computing.

PauseData

#How does Python call low-level C functions from high-level APIs?

• There are three main routes: the C API, the ctypes module, and specialised extension modules. The C API is the fundamental interface, interacting directly with Python objects and the interpreter.
• Python uses reference counting for automatic memory management. C extension code must manage reference counts carefully: increment (Py_INCREF) when it takes ownership, and decrement (Py_DECREF) when it gives ownership up. Getting this wrong causes memory leaks, dangling pointers or objects being freed too early.
• For CPU-intensive C extensions, the GIL can be released explicitly (Py_BEGIN_ALLOW_THREADS) during long-running C computations, and reacquired (Py_END_ALLOW_THREADS) before touching Python objects again. That allows real parallelism for C code that doesn't need the Python API, even inside a multi-threaded Python program.
• C functions designed for Python are compiled into .so (Linux, macOS) or .pyd (Windows) files. When you import the module, PyInit_mymodule is invoked, and it registers the C-implemented functions and data structures with the interpreter.

PauseRequest or acknowledgmentData

Here is the simplest version of the idea, calling a C library function straight from Python with ctypes:

import ctypes
import ctypes.util

# Load the C standard library (this sketch works on Linux and macOS)
libc = ctypes.CDLL(ctypes.util.find_library("c"))

print(libc.abs(-5))  # calls the C function abs() directly: prints 5

#Where does Python's "slowness" come from?

#What is the GIL, and why does it limit multithreading?

• The GIL is a mutex that protects access to Python's objects, to keep CPython's internal memory management thread-safe. In effect, Python bytecode is executed one thread at a time.
  • It is a single, process-wide lock. Before any thread can execute Python bytecode, it must acquire the lock.
  • When a thread does I/O, or calls an external C library that explicitly releases the GIL, other threads can acquire it. Otherwise, threads yield the GIL periodically, based on a configurable instruction count or time slice.
• It simplifies the implementation of CPython by making object management thread-safe.
• It ensures that C extensions, which often manipulate Python objects directly, can't corrupt the interpreter's state.
• It limits true parallelism for CPU-bound tasks. Since only one thread can hold the GIL at any moment, only one thread can execute Python bytecode at any moment. Even with several CPU cores, a multi-threaded Python program doing CPU-heavy work effectively runs sequentially, with extra context-switching overhead on top. The system scheduler might place Python threads on different cores, but only one of them will be running Python code at a time.

#Why can't CPU-bound Python code scale up?

• Fundamentally, because of the Global Interpreter Lock. For CPU-bound tasks, the GIL means only one thread can hold the lock and execute bytecode at any instant.
  • The GIL's main purpose is to simplify CPython's memory management and keep its internal C data structures and C extension modules thread-safe. This design avoids the complex, fine-grained locking that would otherwise be needed for every Python object.
• In an I/O-bound operation, the Python thread can release the GIL while it waits for the I/O to finish. This makes multithreading effective for improving concurrency in I/O-bound scenarios.
• Strategies that get around the GIL's limits include multiprocessing (which uses multiple CPU cores in parallel), C/C++ extensions, and asynchronous programming.

#Why is GPU code almost unaffected by the GIL?

• The computationally intensive parts are offloaded to external, low-level C and C++ libraries that explicitly release the GIL while they run.
• The GIL is bypassed in four steps:
  1. Delegation to native code. High-level Python code calls into its underlying C or C++ extension.
  2. Explicit GIL release. The extension dispatches the actual kernel to the GPU software, using CPython API functions to release the lock.
  3. Asynchronous GPU execution. The GIL stays free while the GPU computes.
  4. GIL re-acquisition. The extension takes the lock again, to keep access to Python objects and interpreter state thread-safe.
• The GIL mainly acts as a mutex for the interpreter's internal state and for Python bytecode execution. Almost all the work in GPU-accelerated computing is native code running on the GPU, outside the interpreter's control, so it doesn't contend for the GIL. The Python thread just starts the operation and then releases the lock, becoming a waiting observer instead of an active bytecode executor while the GPU works.

#Python and the OS: process, memory, syscalls

#What is a Python program inside the OS?

#To the OS, is a Python program a process or a thread?

• Fundamentally, a process. The OS launches the CPython interpreter as a distinct process. That process loads the script, compiles it into bytecode and executes it.
• A Python program can itself manage multiple threads. What you mostly observe is a single process running the CPython interpreter, which may manage several OS threads internally, with the GIL limiting true parallel execution of Python bytecode.

#Why does the OS need to assign it a PID, file descriptors, and a heap?

These are fundamental resources and identifiers that every OS process needs:

The heap matters most for Python:

• It is crucial for data structures whose size isn't known at compile time, or whose lifetime must outlast a single function call.
• Nearly all Python objects (numbers, strings, lists, dictionaries, class instances, functions and modules) are allocated on the heap.
• The memory management system works mainly on this heap space, managing the lifecycle of those objects.

#When does Python trigger a system call (syscall)?

• A syscall is the programmatic interface the OS kernel offers to applications, for accessing privileged resources that user-space programs can't touch directly.
• Python code doesn't invoke the kernel directly. Built-in functions such as open() or socket.send() go through the CPython interpreter, then the C standard library, and finally a syscall.
• Common scenarios are file I/O, network communication, process and thread management, memory allocation, system information and time.
• Syscalls are relatively expensive, because of the overhead of a context switch from user mode to kernel mode.

#How does Python use memory and the file system?

#When Python opens a file, how does it actually call open?

There are three layers between your code and the kernel:

• High-level interface: the built-in open() returns a file object, which hides the complexity of I/O.
• Intermediate layer: CPython's io module and its C functions.
• Kernel gateway: the fundamental mechanism through which user space requests a service from the OS kernel.

In order, for file_obj = open('my_file.txt', 'w'):

1. Python's built-in open() is called.
2. CPython's C-level file I/O takes over. Inside the C implementation, functions like file_open_impl() (from Modules/_io/fileio.c or similar) are invoked.
3. The libc wrapper and the syscall. The C standard library (libc, for example glibc on Linux) acts as a wrapper. Its open() function doesn't implement file opening itself. It prepares the arguments and executes the actual system call.

PauseData

#If there's a file permission error, is it blocked by the OS or by Python?

• Fundamentally, by the OS. High-level file I/O operations are wrappers around system calls.
• The kernel does the access control checks, such as comparing the effective user ID (EUID) or group ID (EGID) against the file's ownership.
• If the required permissions aren't met, the request is denied at that core level. The OS kernel is the ultimate arbiter of resource access.

The flow of control, step by step:

1. Python call. Your code calls a file operation, such as open().
2. CPython wrapper. The interpreter's C implementation translates this into a C function call.
3. System call interface. That C function makes the actual system call to the kernel.
4. Kernel enforcement. The kernel checks permissions. If they're denied, it returns an error code.
5. Python exception. The interpreter receives the error code and raises a PermissionError.

#How do Python's memoryview and buffer map to physical memory?

1. Object allocation. A Python object (for example bytearray(100)) requests memory from the OS through C functions such as malloc. The OS allocates virtual memory pages to the process.
2. Buffer protocol exposure. The object implements the buffer protocol, which provides a Py_buffer structure containing the virtual address of its data.
3. Memoryview creation. A memoryview is created from the object and obtains the Py_buffer details, including that virtual address.
4. Data access. When the memoryview is used to read or write data (for example mv[0]), it computes a virtual address. The MMU (memory management unit) then translates that virtual address into a physical address on the fly, giving direct access to the data in physical RAM.

buf = bytearray(100)   # the object owns a block of memory
mv = memoryview(buf)   # a zero-copy view onto that same block (via the buffer protocol)
mv[0] = 255            # writes straight into buf's memory, with no copy
print(buf[0])          # prints 255

#Multiprocessing and threading: how does Python use multiple cores?

#Why fork a new process instead of opening a thread?

• The reason comes straight from the GIL. Each new process has its own memory space and its own GIL.
• That lets Python bytecode run on a separate CPU core, achieving true parallelism. The multiprocessing module typically does this with the fork() system call.
• Threads are lighter and processes are heavier, but the choice is really dictated by the workload: CPU-bound tasks benefit from multiprocessing, while I/O-bound tasks use threading effectively, or can use asyncio for greater efficiency.

from concurrent.futures import ProcessPoolExecutor


def burn(n: int) -> int:
    return sum(i * i for i in range(n))  # pure-Python, CPU-bound work


if __name__ == "__main__":  # required when processes are started with "spawn" (Windows, macOS)
    with ProcessPoolExecutor() as pool:  # each worker is a separate process with its own GIL
        # four jobs, which can run in parallel on separate cores
        print(list(pool.map(burn, [5_000_000] * 4)))

#How does the OS allocate CPU to multiple Python processes?

• The OS allocates CPU to Python processes through its kernel-level scheduler.
• Each Python process is an independent execution unit, a distinct process from the OS's point of view. The scheduler handles all the processes competing for CPU time.
• Because the interpreter inside each process runs independently, the scheduler aims to maximise CPU utilisation and meet response-time requirements by sharing CPU time among the active processes.
• Scheduling uses preemptive algorithms together with time slicing, which gives each process a time slice (a quantum), and context switching, which saves the state of the running process and loads the state of the next one.
• For Python, multiprocessing is the standard way to use CPU cores while getting around the GIL. Multiple Python processes each have their own GIL and can run Python bytecode in parallel on different cores. They can be CPU-bound or I/O-bound: I/O-bound processes often yield the CPU, while CPU-bound processes use their full time slice.

#Do Python subprocesses share GPUs?

• Yes, but how they share and interact is different from the CPU, and it needs careful thought about the GPU architecture and driver mechanisms involved.
• Subprocesses generally request and are given their own separate segments of VRAM.
• Python itself can't manage GPU hardware directly. It is an interface to low-level libraries, which make the actual API calls to the GPU driver to control the hardware.
• Advanced mechanisms such as explicit resource partitioning and inter-process GPU memory sharing also exist, for coordinated GPU usage.

PauseData

#Python and the CPU: where does computation happen?

#How does the CPU handle Python's inference logic?

#How does the CPU execute NumPy and PyTorch operations?

• Through the libraries from who is actually running the code: BLAS and LAPACK for NumPy, and MKL, OpenBLAS and oneDNN for PyTorch. On the CPU they add two levels of parallelism:
  • Inside a core, vector (SIMD) instructions process several numbers per instruction (next question).
  • Across cores, one large operation is split over several threads, which is called intra-op parallelism and usually runs on OpenMP. torch.get_num_threads() shows how many threads PyTorch uses, and torch.set_num_threads(n) changes it.
• Python makes one call per operation, so the interpreter's overhead stays out of the numerical loops.

#Are these operations pure Python, or do they use SIMD / AVX instructions?

• The actual numerical computation in NumPy and PyTorch is offloaded to compiled, optimised routines that talk directly to the CPU's hardware features, including its SIMD and AVX instruction sets, for maximum performance.
• SIMD (Single Instruction, Multiple Data) is a class of instruction sets in which one CPU instruction performs the same operation on many data points at once.
  • This is fundamental for scientific computing and machine learning, where operations like vector additions, matrix multiplications and element-wise functions are applied to large arrays or tensors.
• AVX (Advanced Vector Extensions) is a set of SIMD instruction extensions developed by Intel (and supported by AMD). It widens the vector registers and adds new instructions for floating-point and integer operations.
  • These extensions are critical for performance in highly parallelisable workloads, which are everywhere in deep learning and high-performance computing.

#How does PyTorch's CPU backend (MKL, OpenBLAS) get used by Python?

• Using MKL or OpenBLAS is a prime example of a Foreign Function Interface.
• PyTorch has a split architecture: a high-level Python API for users, and a low-level C++ core for computation. That core is called ATen.
  • ATen (A Tensor Library) is PyTorch's C++ tensor library. It provides the foundational data structures and operations, and acts as an abstraction layer for tensor manipulation that is independent of the hardware (CPU or GPU).
  • For CPU-bound operations, ATen doesn't re-implement optimised linear algebra from scratch. It links against highly tuned external libraries such as Intel MKL or OpenBLAS. These provide optimised BLAS and LAPACK functions, the building blocks for most deep learning operations (matrix multiplication, convolutions, reductions).

The path of a single call looks like this:

1. Python API call
2. Data bridge between the Python object and the C++ tensor
3. Dispatch to the ATen C++ kernel
4. Backend function invocation (MKL or OpenBLAS)
5. Optimised CPU execution
6. Result packaging back into a Python object

PauseRequest or acknowledgmentData

#Are Python threads truly parallel?

Not for CPU-bound Python code, because of the GIL. Threads give you concurrency (taking turns and overlapping waits), but not parallelism (running at the same instant). Which tool gives which:

#What operations does the GIL prevent from running in parallel?

• Operations that involve intensive computation purely within Python, without explicitly or implicitly releasing the GIL, are serialised.
• Examples are pure-Python arithmetic, manipulation of standard data structures, and user-defined functions that do heavy computation without delegating to external C libraries or doing I/O.
• Any operation on the Python object model that is guarded by the interpreter's internal locks.

#Which built-in operations automatically release the GIL?

• Blocking I/O. Reading and writing files and sockets, select, time.sleep() and waiting on a subprocess all release the GIL while the thread waits for the operating system.
• Some CPU-heavy built-ins. hashlib releases it while hashing large buffers, and zlib, bz2 and lzma while compressing.
• Native libraries. Most NumPy operations on large arrays, and most PyTorch operations, release it inside their compiled code.
• Waiting on locks. A blocked threading.Lock.acquire() or queue.Queue.get() releases it while it waits.
• What never releases it: pure-Python loops, arithmetic on Python objects, and work on lists and dictionaries.

#Can multiprocessing bypass the GIL?

• Yes. Each process has its own interpreter, its own memory and its own GIL, so CPU-bound Python code runs in parallel on several cores.
• The price. Arguments and results are pickled and copied between processes, each process holds its own copy of the interpreter and of any data it loads, and starting a process takes much longer than starting a thread. For large arrays, multiprocessing.shared_memory avoids the copies.
• That is why data loaders use worker processes. PyTorch's DataLoader(num_workers=4) runs four processes that prepare batches in parallel while the main process feeds the GPU.

#Where does asyncio fit in?

• asyncio gives concurrency on a single thread. Functions declared with async def are coroutines, and an event loop runs them one at a time. When a coroutine reaches an await on something that isn't ready yet (a network reply, a timer), it pauses, and the loop runs another one.
• Nothing runs in parallel, so the GIL never comes into play. The gain is that thousands of waits can overlap without thousands of threads.
• This is how LLM servers handle many users at once. vLLM's API server is an asyncio application (post 05), and the scheduler project in post 06 is built the same way.
• The catch: a coroutine that computes for a long time without awaiting blocks the whole loop. Blocking work belongs on a thread (asyncio.to_thread) or in another process.

import asyncio


async def fetch(name: str, seconds: float) -> str:
    await asyncio.sleep(seconds)     # stands in for a network call: the loop runs others meanwhile
    return f"{name} done"


async def main():
    # three one-second waits overlap on one thread: about 1 second in total, not 3
    print(await asyncio.gather(fetch("A", 1), fetch("B", 1), fetch("C", 1)))


asyncio.run(main())

#How Python calls the GPU

#How does Python send data to the GPU?

#What exactly does .to("cuda") do? How fast is it? Is it blocking?

• .to("cuda") is a method in deep learning frameworks like PyTorch. It transfers a tensor or model from the CPU's memory to the GPU's memory, so that NVIDIA's GPUs can apply their parallel processing power through CUDA.
• Its speed depends on the size of the data, the system bus bandwidth, the memory type and the CUDA environment setup.
• By default the copy is blocking from the CPU's point of view: the Python thread waits until the transfer has finished (non_blocking=False). Passing non_blocking=True, with the source in pinned memory, lets the copy overlap with other work.
• What is asynchronous by default is launching GPU work. A kernel launch is queued on the GPU and returns immediately, so the Python script carries on with the next lines of code on the CPU.
• Synchronization happens with torch.cuda.synchronize(), and whenever data is copied back from the GPU to the CPU.

import torch

x = torch.randn(1000, 1000)  # a tensor in CPU (host) memory
x_gpu = x.to("cuda")         # copy to GPU memory: blocks this Python thread by default
y = x_gpu @ x_gpu            # queues a kernel on the GPU and returns immediately
torch.cuda.synchronize()     # wait until the GPU has finished everything queued

#Who decides which GPU to use? How are multiple GPUs managed?

GPU device selection follows a hierarchy of mechanisms, for both flexibility and control:

• Default behaviour. By default, frameworks such as PyTorch and TensorFlow target the first available GPU, usually identified as cuda:0 (device 0).
• Environment variables. System-level variables influence GPU visibility and ordering without touching the code. The best-known is CUDA_VISIBLE_DEVICES. Setting CUDA_VISIBLE_DEVICES="1,3" makes only the GPUs with physical IDs 1 and 3 visible to the application, which then sees them as device 0 and device 1.
• Explicit code. Developers can assign tensors, models or operations to specific GPUs in code, with .to("cuda:1") or a torch.device("cuda:2") object.

import os

os.environ["CUDA_VISIBLE_DEVICES"] = "1,3"  # must be set before CUDA is initialised

import torch

dev = torch.device("cuda:1")  # logical GPU 1, which is physical GPU 3 after the line above

Using several GPUs well is critical for scaling deep learning workloads, and there are several strategies:

#Do you need to manually free CUDA memory? What does PyTorch handle for you?

• In PyTorch, you generally don't need to free CUDA memory manually. PyTorch has memory management that handles GPU allocation and deallocation for you.
• It uses a caching allocator for CUDA memory, a fundamental design choice that improves performance and simplifies memory management.
  • How allocation works. When you request memory (for example by creating a tensor on the GPU), the allocator first checks its cache for an already allocated but currently unused block that is big enough. If it finds one, it reuses it. If not, it asks the CUDA driver for a new block.
  • How deallocation works. When a tensor goes out of scope or is deleted, its GPU memory is not immediately returned to the CUDA driver. It is marked free and goes back to PyTorch's internal cache, ready for future allocations without the overhead of a CUDA driver call.

PauseData

PyTorch handles several critical aspects of CUDA memory for you:

Manual intervention is rarely needed, but there are a few cases where you interact with the cache:

print(torch.cuda.memory_allocated())  # bytes currently used by live tensors
print(torch.cuda.memory_reserved())   # bytes PyTorch holds from the driver (includes free cached blocks)

torch.cuda.empty_cache()  # hand unused cached blocks back to the CUDA driver

• torch.cuda.empty_cache() clears PyTorch's internal cache and releases all unused cached memory to the CUDA driver. It's useful when you suspect fragmentation, after deleting large intermediate tensors, or to get an accurate reading of current GPU memory use. It does not free memory held by active tensors. It only empties the cache of blocks that were allocated but are no longer referenced.
• torch.cuda.memory_allocated() and torch.cuda.memory_reserved() show how much memory PyTorch is actively using and how much it has reserved from the CUDA driver, including cached free blocks.

At a low level, the caching allocator still talks to the CUDA Driver API:

• Allocation: cudaMalloc(void** devPtr, size_t size) allocates size bytes of linear memory on the device and returns a pointer in devPtr. PyTorch's allocator calls this when its cache is exhausted.
• Deallocation: cudaFree(void* devPtr) frees the memory that devPtr points to. PyTorch's allocator calls this when empty_cache() is invoked, or when PyTorch's overall memory request passes a threshold (less common in normal operation).

#How is computation on the GPU triggered?

#Does calling model() execute on the GPU immediately, or is there lazy evaluation?

• A model() call on the GPU from Python is a nuanced blend of graph-based execution and optimised kernel launches, not a single atomic step.
• In an immediate (eager) execution model, each operation runs as soon as it is called, and the result is available straight away. This is more interactive and imperative.
• In lazy evaluation, operations define a computational graph without executing it right away, and the computation is deferred until something triggers it.
• By default, PyTorch runs in eager mode: the operations of the forward pass are performed one after another.
• TensorFlow 2.x also defaults to eager execution, but strongly encourages tf.function, which compiles Python code into a static TensorFlow graph. The graph is traced once, and later calls run the compiled graph directly on the GPU, which brings significant gains from global optimisations, memory pre-allocation and less Python overhead.

#How is a GPU kernel registered and launched?

1. Definition. Kernels are typically written in low-level, parallel-aware languages such as CUDA C/C++ for NVIDIA GPUs, or OpenCL C.
2. Compilation. The kernel source goes through a specialised compilation process that turns it into a device-specific binary format.
3. Loading. The GPU driver loads the compiled kernel library into GPU memory, which makes the kernel function accessible and ready for the device to run.
4. Launch. From the CPU (the host), a specific API call starts the kernel's execution on the GPU (the device).
5. Arguments. Pointers to data in device memory, and scalar parameters, are passed from the host to the kernel function running on the device.
6. Queueing. The runtime API call becomes commands that the GPU driver submits to the GPU's internal command queue, for asynchronous execution.

The queue is a CUDA stream. Work on one stream runs in order, while work on different streams can overlap, for example a copy on one stream while a kernel runs on another. PyTorch uses one default stream per device unless you create more.

PauseDataRequest or acknowledgment

#How can you check which kernel an operation uses?

• This means looking into the runtime environment, using framework-specific introspection tools or low-level profilers.
• PyTorch: the most comprehensive tool is the Torch Profiler, which captures detailed execution traces with names and durations.
• TensorFlow: the TensorBoard profiler captures execution traces on both CPU and GPU, with visualisations of the operations, launches and usage.
• Low-level NVIDIA profilers (Nsight Systems and Nsight Compute) give a detailed timeline of CPU and GPU activity, including every CUDA kernel launch, API call, memory transfer and synchronization event, plus kernel-level analysis.

from torch.profiler import profile, ProfilerActivity

# Record both CPU and CUDA activity while the model runs
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    model(x)

# A table of the operations and kernels that ran, with their timings
print(prof.key_averages().table(row_limit=10))

#How do the CPU and GPU synchronize?

#Is .cuda() and .cpu() data transfer synchronous or asynchronous?

• Synchronous by default, like .to("cuda") above: the CPU thread waits until the copy has finished.
• .cpu() also has to wait for every kernel that produces the tensor, so it is a synchronization point as well as a copy.
• .cuda(non_blocking=True) and .to(device, non_blocking=True) make the copy asynchronous. From the host it overlaps with other work when the source is in pinned memory. Toward the host, the values aren't safe to read until the stream has been synchronized.

#When does PyTorch force synchronization?

• Whenever the CPU needs the results of GPU computation, or when the order of operations across the host and device must be strictly enforced for correctness or resource management.
• Implicit synchronization happens through a barrier that blocks the CPU until the relevant GPU computation is done. Common triggers are copying a result back to the CPU (.cpu()), reading a single value with .item(), and printing a GPU tensor.
• Explicit synchronization is where developers insert synchronization points by hand, for fine-grained control, performance profiling or debugging.

#What is torch.cuda.synchronize(), and when should you call it manually?

• It is a blocking call that makes sure all GPU operations on the current CUDA device, started by the host (CPU), have completed before the CPU moves on.
• Its fundamental purpose is to enforce synchronization between CPU and GPU. GPU operations are asynchronous: the CPU queues operations and carries on without waiting for the GPU to finish.
• Call it manually to measure the execution time of GPU operations accurately.
• Calling it can also force an error to appear immediately at the point of synchronization, which makes the origin of the problem easier to find.
• Call it to access on the CPU data that was just modified on the GPU: synchronizing guarantees that the GPU has finished writing the data to memory before the CPU tries to read it. This is especially true when using .cpu() on a tensor that has just been computed on the GPU.

import time

torch.cuda.synchronize()        # make sure earlier GPU work has finished
start = time.perf_counter()
y = model(x)                    # asynchronous: returns before the GPU is done
torch.cuda.synchronize()        # wait for the GPU, or we would only time the launch
print(time.perf_counter() - start)

#Summary

A Python program runs as source, tokens, an AST, bytecode and finally the PVM, all inside CPython, which is itself a C program. With torch and numpy, Python only orchestrates: the math runs in compiled C, C++, Fortran and CUDA libraries. That is also why the GIL, which lets one thread run bytecode at a time, slows down pure-Python threads but not GPU work. The operating system sees a process with a PID, a heap and file descriptors, and every file or network access is a syscall that the kernel checks. On the CPU, libraries use SIMD instructions and several threads per operation. On the GPU, copies block by default, kernel launches are asynchronous, PyTorch caches freed memory, and torch.cuda.synchronize() is how you wait for the results.

#Concepts at a glance

#Test yourself

Next in the series: 02 · Neural Network Fundamentals.