Raph Levien: GPU Compute, Functional Design, and a 2D Renderer

GPU compute is a deep and technical subject, so this is a taste rather than a complete guide. I will explain the hardware, suggest a functional way to structure GPU programs, and use my work on a high-performance 2D renderer as an example.

The final part concerns deployment: once you have written a GPU program, getting it to run across hardware is still difficult. This talk reflects the hardware and evolving APIs available around 2020.

As Dennard scaling slowed, increasing single-core speed stopped providing the same route to throughput. Parallelism became increasingly important. GPUs provide substantially more of it than CPUs, although the useful result depends on the workload.

A historical chart of theoretical floating-point throughput shows high-end GPUs roughly an order of magnitude above CPUs. Levien also estimates a similar order-of-magnitude advantage per dollar and per watt in the examples he discusses, motivating workloads such as machine learning.

Chip photographs show how much area can be devoted to the GPU and its shared memory infrastructure. But GPU designs differ considerably; the hardware tour is intentionally simplified and focuses on desktop-class devices rather than mobile GPUs.

In the illustrated Intel execution unit, several execution threads have their own program counters and register files. Each thread has multiple lanes. An arbiter chooses ready work and matches it to arithmetic, branch, and load/store units.

A CPU invests heavily in improving a single thread through pipelines, register renaming, and superscalar execution. The GPU emphasizes throughput: while one execution thread waits on memory, others can use the arithmetic units and hide latency.

A load sends a request to memory and the thread waits for the response; the arbiter keeps other ready threads moving. Levien gives an approximate two-nanosecond latency for operations within the illustrated execution unit, emphasizing that these are round numbers.

An execution thread processes a SIMD vector, with width depending on the hardware. The shader source usually looks like code for an individual thread, even though the hardware executes multiple lanes together; the word thread is used at different levels.

If all lanes take the same branch, execution is straightforward. If some take one branch and others the other, the hardware executes both paths with masks. This divergence leaves inactive lanes idle and increases the work performed relative to useful work.

Execution units are grouped into larger structures sharing resources such as instruction caches and memory access. A texture sampler can provide fast lookups and hardware interpolation; the exact organization and latency vary by device.

Moving farther out through the hierarchy reaches larger shared structures and main memory. More distant access costs more latency. Adding execution resources at different scales produces devices with different power and throughput characteristics.

Compute work is dispatched as one-, two-, or three-dimensional grids of thread groups. The programmer must map the problem into that grid and choose group sizes that expose enough parallel work for the hardware.

Although shader code looks imperative, I find it useful to think functionally. A thread group reads input buffers, computes and communicates through group-local memory, and writes output buffers. Each output is a function of inputs; the dispatches form a dependency graph.

Machine-learning workloads map naturally to graphs of operations and grids of work, which helps explain why GPUs suit them. For other domains the question is whether we can find an equally effective decomposition into dispatches and threads.

Performance is strongly constrained by the memory hierarchy. Registers near arithmetic units have high bandwidth and low latency but little capacity. Global memory provides much more storage at higher latency and lower relative bandwidth.

Thread-group shared memory sits between those extremes and must be used deliberately. In the examples on the slide, capacity is limited to tens of kilobytes per group. Organizing work to reuse nearby data can reduce costly global transfers.

My motivation is 2D rendering. Existing renderers such as Skia and Direct2D commonly map a scene to graphics primitives and use fixed-function pipelines. They work, including on older hardware, but complex blending and intermediate buffers can cost bandwidth.

A compute-based renderer can instead do intermediate blending in compute and write a final pixel once. That is not free: the GPU must traverse the scene data structures uploaded by the CPU. My prototype, piet-metal, explores that trade-off.

In a simplified model, the scene is a list of objects, each able to render a color at a given coordinate. Composite the colors using the associative Porter-Duff over operator. A functional program maps over pixels and folds each pixel’s object colors.

This naive formulation has excellent parallelism: pixels are independent, and an associative fold offers further parallelism. It also writes each final pixel only once. Yet every pixel must read every scene object, most of which do not contribute to it.

Bounding boxes reveal that most objects affect only a small part of the image. A sequential CPU-style program visits each object once, then renders only the pixels in its box. It reduces object reads and unnecessary work, but the simple formulation has poor parallelism.

Its write traffic also depends on the scene. With many overlapping or blended objects, the renderer repeatedly reads and writes the same pixels in memory. Simply adding a per-pixel bounding-box filter to the naive parallel version does not solve the repeated-read problem.

Instead, divide the image into tiles and use two stages. A 16-by-16 tile has 256 pixels, so filtering once per tile rather than once per pixel can greatly reduce object reads. Tune tile size to the hardware and render the pixels in a second stage.

The functional formulation maps over tiles, filters objects whose bounding boxes intersect each tile, renders the pixels using those objects, and stitches the tiles back into an image. Parallelism and final-write behavior remain good while read work is reduced approximately by the tile area.

On the GPU this becomes two dispatches. The first builds a command list for each tile; the second reads that list to render pixels. Levien compares it to specializing a scene program into per-tile bytecode and then interpreting it.

There is still room to improve work utilization. Consider a 32-by-32 chunk of objects and tiles. A straightforward lane tests each object against its tile and writes an intersecting object into the output list.

Across the SIMD lanes, each iteration reads the same object and tests it against many tiles. That uniform read can be efficient, but most objects do not intersect most tiles. Masked writes and repeated iterations leave many lanes doing little useful work.

A more cooperative design lets lanes read different bounding boxes in parallel and compute intersection bit masks. A transpose exchanges information between lanes: masks indexed by object and containing tiles become masks indexed by tile and containing objects.

Each tile can then emit its hits in parallel, iterating only up to the largest number of hits in the chunk. If nothing intersects, the group can skip directly to the next chunk. Bit operations and subgroup shuffles provide the needed cooperation.

This optimization was a design rather than an implemented benchmark at the time of the talk. I expect it to improve performance, but it needs shader code and testing on real hardware before that expectation is established.

Deployment remains a separate challenge. CUDA was an early practical GPU-compute platform; Levien describes OpenCL as losing momentum at the time and notes increasing compute support in graphics APIs such as DirectX 12 and Metal.

He expects Vulkan and WebGPU to grow alongside CUDA and highlights SPIR-V as an intermediate representation. A common intermediate layer can separate source-language tooling from the hardware driver, but abstractions may not expose every newly available hardware feature immediately.

Shader ecosystems historically used platform-specific languages and intermediate forms, such as Metal shading language and HLSL. SPIRV-Cross can translate an intermediate representation back to source for other targets. Levien discusses the then-new WGSL proposal and Tint as part of an evolving WebGPU design.

Vulkan 1.0 exposed compute kernels and group-local memory; Vulkan 1.1 added subgroup shuffle and reduction operations and restricted pointers. Levien highlights Vulkan 1.2’s formal memory model and features such as 64-bit atomics and optional 16-bit floating point.

WebGPU was still a proposal and prototype in this 2020 presentation. The speaker describes the debate over binary versus textual shaders and a proposal to align WGSL semantics with SPIR-V. These statements describe that historical design discussion.

Safety can impose costs such as clearing memory or extra synchronization when buffers cross trust boundaries. Levien also points to native libraries such as wgpu and Dawn, and the Tint compiler, as possible routes to using similar interfaces outside the browser.

Higher-level GPU languages offer different approaches. Halide targets image processing; Futhark explores an ML-family functional language for GPUs; Co-dfns takes inspiration from APL and array programming; Julia offers an interactive workflow, often using CUDA at the time.

Levien also discusses MLIR and the possibility of lowering machine-learning computations toward SPIR-V and Vulkan. He hopes a more solid intermediate representation, memory model, and shared tests will reduce the friction of deploying compute workloads.

His prediction is that better tools and languages could make GPU throughput useful for more than games, machine learning, and mining. He wants programmers to express an intentional mapping to hardware with less friction, rather than rely on a magically optimal compiler.

An audience member asks whether sorting objects could increase the number of chunks skipped after bounding-box tests. Levien says it might, but the benefit must be weighed against the cost of sorting. Reordering nonintersecting objects and other spatial structures are further possibilities to explore.

Another question asks how to choose an optimal thread-group size across different hardware. Levien distinguishes programmer-controlled thread groups from hardware/compiler-dependent subgroups. At the time, controlling subgroup width was not uniformly supported.

In practice, he says, teams may benchmark multiple kernel versions and use hardware queries and heuristics to choose among them. He does not claim to have a general automatic solution; portable performance remains a difficult engineering problem.

Finally, asked whether graphics and machine learning hardware will converge, Levien expects more specialized low-precision tensor hardware for machine learning, which may not help other domains equally. Better tools and deployment interfaces, however, can benefit both graphics and machine-learning workloads.