Kernel Development

This is the smallest cluster and the deepest one: writing and profiling the CUDA and Triton kernels that everything else in this blog eventually calls. It's for engineers who've hit a wall that quantization or a different serving engine can't fix, and need to know what's actually happening on the streaming multiprocessor.

The posts here cover kernel authoring (OpenAI's Triton for writing GPU kernels in Python, CUDA 13's new Tile programming model with the cuTile DSL) and profiling (Nsight Compute roofline analysis, Nsight Systems, PyTorch Profiler with HTA) alongside the specific kernel libraries and attention variants that define current inference performance: FlashAttention through its four generations, FlexAttention's compiled custom masks, Multi-Head Latent Attention's roughly 98% KV cache compression, and Liger Kernel's fused Triton patches for training.

The pillar post, FlashAttention-4 on GPU Cloud, covers the newest generation's tile-based kernels for Blackwell, benchmarked against FA3 with migration steps for vLLM and SGLang. If you're new to kernel-level work, start with the Triton or profiling posts to build a mental model before a specific kernel's internals.