vLLM · CUDA · Triton · Runtime Reliability

Inside LLM Inference Runtime

A source-to-system study of high-performance LLM serving: following a request through vLLM, crossing the Python/CUDA boundary, and rebuilding FlashAttention from a mathematical reference into a Triton kernel.

Part 1

Inside vLLM: Following One Request from the API to GPU Execution

The V1 request lifecycle, scheduling, KV-cache management, model execution, and output delivery.

Part 2

Inside vLLM CUDA Kernels: KV-Cache Writes and Paged Attention

GPU execution fundamentals, layouts, vectorized copies, coalescing, tiles, and FlashAttention-style paged attention.

Part 3

Building a Verifiable FlashAttention Forward from PyTorch to Triton

Explicit attention, tiled online softmax, a fused Triton kernel, and correctness and performance measurements.

Built and measured

FlashAttention Triton lab: an original contiguous-attention forward kernel, a tiled PyTorch oracle, 63 correctness cases, and 56 published latency/memory records.
vLLM runtime diagnostics lab: fault-injection, process-exit, memory-boundary, preemption, and stability experiments for inference-runtime failures.

Upstream vLLM work

PR #49869: Fix GLM-OCR MTP weight loading — merged into vLLM after model-loading regression tests and independent GPU reproductions.
PR #52178: Propagate fatal EngineCore shutdown — process-lifecycle fix with signal controls and single-/multi-GPU failure injection.
RFC #54229: Privacy-safe runtime incident snapshots — a bounded, opt-in evidence artifact for fatal EngineCore failures, with a published schema and evaluation protocol.
Contribution fork