Sylvain Gugger on Accelerating ML Models and Reproducing Results

Ron Minsky introduces Sylvain Gugger, an ML engineer at Jane Street, former maintainer of Hugging Face Transformers, creator of Accelerate, and co-author with Jeremy Howard of Deep Learning for Coders with fastai and PyTorch. Gugger previously taught mathematics, moved to the United States in 2015, and discovered machine learning through online courses in 2017.

Gugger took Howard's fastai course, contributed to its forums and library, and joined a team competing to train a vision model to a target accuracy quickly. They led for much of the contest before Google released TPUs and surpassed their time. He says fastai's educational mission includes giving domain experts enough ML knowledge to improve models in fields such as radiology.

One contest improvement was the learning-rate schedule. Gradient descent updates weights using gradients, and the learning rate determines step size. Instead of training a long time at a small rate and then reducing it, the team warmed up from a low rate to a higher one, trained briefly and decreased it toward the end, reaching the target accuracy faster.

Gugger's intuition is that a randomly initialized model may sit in a steep, irregular part of the loss landscape: large initial steps can destabilize it. Small initial steps help, then larger steps traverse the landscape and can skip local minima, while smaller final steps refine the result. Both note that a two- or three-dimensional picture is an imperfect analogy for millions of parameters.

The second improvement was gradual resizing. A convolutional network can operate on different image sizes, so training began with smaller 128-by-128 images and increased resolution later. Early iterations were cheaper because filters processed fewer pixels. Gugger says that schedule became widely adopted, while gradual resizing remained less common.

After working on fastai's library and courses, Gugger co-wrote the book as a reference for people who prefer reading. He joined Hugging Face in 2020, maintained open-source libraries, developed tutorials and a course on Transformers. He describes its hub as a place to share model weights and its libraries as the code needed to instantiate those models.

Hugging Face researchers found a monolithic trainer increasingly difficult to modify as flags accumulated. Gugger created Accelerate so researchers could write their own training loops and make a few changes to run them on different hardware. The goal was flexibility with minimal intrusion, in contrast with a black-box trainer.

Data parallelism splits a dataset across GPUs, which compute portions of training in parallel. Accelerate handles boilerplate and differences between GPU, TPU and CPU environments, as well as mixed precision. Ron notes that model parallelism is also needed when weights no longer fit on one GPU.

Splitting layers across devices makes a large model fit but can leave later GPUs waiting. Pipeline parallelism overlaps stages on successive data portions, including forward and backward work. Tensor parallelism splits the weights themselves and needs especially fast communication to combine the results of matrix operations.

Ron and Gugger discuss networking as a core part of distributed training. RDMA can move data from the network interface directly into memory, potentially GPU memory, without CPU involvement in the transfer. Fast interconnects and physically close GPUs reduce communication costs; Gugger describes emerging cabinet-scale systems as an example.

At Jane Street researchers bring Gugger models that train slowly. He profiles them to locate bottlenecks. PyTorch's eager execution is flexible, but code that appears harmless can be inefficient, for example when an operation requires CPU-GPU synchronization or moves work away from the GPU.

The CPU normally dispatches GPU operations asynchronously and can stay ahead. When Python needs a result from the GPU, the CPU waits, and the GPU can subsequently sit idle waiting for more work. A loop that repeatedly launches tiny operations and synchronizes can be particularly costly; kernel launches themselves have overhead.

Ron distinguishes two meanings of vectorization: in low-level CPU programming it may mean using SIMD instructions, while in Python it often means moving the loop into native code such as NumPy. In both settings avoiding slow interpreted loops matters, but GPU work also needs to avoid unnecessary synchronization.

Ron asks why PyTorch succeeded despite the appeal of compiled computational graphs in TensorFlow and JAX. Gugger attributes much of its success to researchers' desire to experiment and iterate flexibly; an idea can be optimized later if it proves valuable. Asynchronous dispatch can still be fast when the code avoids synchronization.

With faster GPUs, Python dispatch can struggle to launch kernels quickly enough. `torch.compile` captures a computational graph and can fuse consecutive operations into larger kernels. Fusion reduces launch overhead and can keep intermediate values in registers instead of writing them to GPU memory and reading them back.

Gugger describes Triton as a Python-like domain-specific language for generating GPU kernels and a component used in this compilation approach. It is especially useful for matrix-style workloads, where tensor cores offer high throughput. When a workload does not fit its supported patterns, compilation and debugging can be difficult.

CUDA is the low-level NVIDIA programming environment described in the conversation, with explicit parallel kernels and C-like hazards such as pointer mistakes and undefined behavior. GPU threads run in groups called warps, and blocks share local memory. Registers and rapidly scheduled warps allow a GPU to hide some memory latency by running other work while a group waits.

Between low-level CUDA and high-level frameworks there are several languages and tools. Gugger mentions Triton, Numba and Google projects as examples. He and Ron discuss Mojo cautiously as an emerging Python-like language, noting that their understanding and its GPU support are prospective at the time of the recording.

For a slow model, Gugger profiles repeatedly and looks for CPU-GPU synchronization. Some synchronization can be removed by expressing the computation differently. A subtle example is checking in Python whether a GPU-computed loss is NaN: the CPU must wait for the result before choosing a branch.

One possible approach to the NaN check is to run it in another thread while the main thread continues, accepting that a few extra iterations may run before the check stops training. After removing synchronization, compiling can enable kernel fusion. These are performance choices to be measured, not a promise that every model benefits equally.

Mixed precision can accelerate training: matrices may use 16-bit floating-point values while accumulation uses 32-bit precision. Gugger describes potential two- to fourfold speedups depending on the problem and hardware, including use of tensor cores and bfloat16. He and Ron treat still lower precision formats as emerging rather than assuming they work for everything.

Ron asks whether users can easily see when code is slow. Gugger says explicit control over data placement in PyTorch helps, but profiling is still needed to find hidden synchronization. The team is trying to make profiling and traces easier to access and identify unusually slow steps automatically.

When a researcher's idea cannot be expressed efficiently with existing operations, the team may need a custom Triton or CUDA kernel. CUDA graphs offer another way to reduce repeated kernel-launch overhead by recording and replaying a sequence. Unlike fusion, graphs do not eliminate the intermediate memory writes and reads between kernels.

CUDA streams allow independent work to be scheduled concurrently, such as transferring the next batch while computing on the current one. Gugger cautions that streams are a hint and hardware may choose not to overlap work; trying to run very many independent streams is not necessarily efficient.

Feeding GPUs efficiently matters as much as optimizing kernels. Variable-length samples can waste memory if short sequences are padded to the longest one. Packed representations can reduce that waste, but distributed training also needs to balance work across GPUs so short batches do not wait at a synchronization barrier for a long one.

Grouping samples by length helps balance the load but can reduce the randomness of a shuffle and potentially affect training accuracy. The speakers describe this as a performance-versus-training tradeoff. Gugger also notes that PyTorch's eager execution is convenient when batch shapes vary, while compiled systems may require more work to handle changing shapes.

Ron asks how Jane Street's internal tooling compares with open-source ecosystems. Gugger says profiling has sometimes revealed simple changes that made a training run four times faster, but sees a need for more modular training infrastructure. Researchers want composable building blocks they can modify rather than a giant trainer with many flags.

Ron distinguishes general organizational AI applications, external image or text data used for trading, and models trained directly on market data. In the first two cases existing architectures can be useful. Direct market-data modeling has a different shape: the data is noisy, with a weak signal that may nevertheless support valuable trading decisions.

Markets adapt as participants exploit patterns, so a model useful a few years ago may no longer work. Gugger says market data can amount to terabytes per day, and models may be smaller than the largest language models. The resulting compute-to-data ratio makes efficient loading and hardware utilization important, often requiring custom models and research.

Inference requirements vary widely. Ron gives examples from extremely low latency, including roughly 100 nanoseconds in some applications, to microseconds, milliseconds or even hourly predictions in others. Gugger adds that throughput matters too: keeping up with a high rate of events can require only a few microseconds per event even when each prediction need not be returned immediately.

Jane Street develops custom loading utilities and models for these needs, while much of the training procedure still uses familiar stochastic gradient descent. Gugger also participates in education: boot camps help traders and researchers learn ML so their domain knowledge can inform model choices, and some go on to train models themselves.

Gugger says an especially difficult lesson is that ML remains empirical. Theory explains some aspects of fitting, but why neural networks generalize so well is not fully understood; a technique that seems likely to fix overfitting must still be tried. He compares the process to cooking: prepare ingredients, let the model train and see how it turns out.

Reproducibility is another hard lesson. A promising run is of limited use if, months later, nobody knows how it was produced. Gugger recommends saving the exact code revision, dependencies, training configuration and hyperparameters such as learning rate and batch size. Ron notes that tools should make discipline easier, even though careful research remains necessary.

Notebook cells can execute in arbitrary order and leave results whose provenance is unclear. GPU parallelism adds another source of nondeterminism: floating-point addition is not associative, and parallel work may complete in different orders. Deterministic execution is possible in some cases but can carry a substantial performance cost.

Ron mentions highly asynchronous approaches such as Hogwild, where concurrent updates can have data races. Such methods can work numerically, but Gugger says the resulting model can be difficult to explain or reproduce. The discussion ends with reproducibility as a practical constraint alongside speed.