Three Ways to Shard AI Models and Their Tradeoffs

There are three broad ways to shard a model computation: split the model weights, split the input activations, or divide the model into stages, known as pipeline parallelism. The speaker says pipeline parallelism helps when a model is too large, but does not itself improve latency.

Sharding the weights, also called model or tensor parallelism, can address model size and help latency. But sharding has a cost, so there is a performance tradeoff.

Input sharding is relatively cheap, but usually does not solve the model-size problem. Attention requires extra care: tokens attend to other tokens, so splitting them introduces additional complexity.

The speaker’s practical advice is to shard only as much as necessary. Sharding brings overhead, but some degree of it can be unavoidable when a model cannot otherwise fit.