Petals: Distributing Large Language Model Inference Across Devices

I introduce Petals as a potentially important development for accessing large language models. My enthusiasm about its significance is a forecast. The goal I describe is to use a model through a distributed network without needing one client device to hold the complete model.

I contrast centralized services such as ChatGPT with models such as LLaMA, BLOOM, and MPT that users could obtain and run. My concern is the cost of hardware for large models: I estimate that a 33-billion-parameter model needed an expensive GPU to achieve reasonable inference speeds. That cost and availability context is from 2023, and licenses differ among models.

Petals offers another approach: distribute inference and fine-tuning across a network. I compare the shared-resource idea to torrents, but this is an analogy rather than evidence that Petals is BitTorrent. A client’s ability to use the system depends on compatible software, networking, and participating servers, not literally every device without conditions.

I describe the model as split into blocks hosted by different participants. Each contributing server supplies a portion rather than needing to run the whole model. The client coordinates work across the network, and a user can also access it from an environment such as a Google Colab session rather than using only a local GPU.

I cite a speed of approximately five to six tokens per second for LLaMA 65B. The figure is a historical reported observation, not a guarantee: network latency, available hosts, model, and configuration can change performance. The table shown separately in the video reports benchmarks for BLOOM-176B under several server and network setups.

A participant can be a client, a server, or both. I describe clients using the network for inference or fine-tuning and servers contributing hardware to run model blocks. A client can use the Petals Python library without personally managing every server, while a group can also create a private swarm.

I speculate that distributing model components might be useful for mixture-of-experts architectures. I also mention a then-circulating description of GPT-4’s internals; the video does not verify those details or demonstrate that Petals supports a particular mixture-of-experts system.

A practical challenge is that the public network depends on people contributing available compute. I imagine idle GPU time as a potential resource, but the video does not measure how many contributors are needed or guarantee that a volunteer network will always have sufficient capacity.

One possible incentive I suggest is rewarding server operators with tokens, perhaps providing priority or higher usage and potentially tradable value. This blockchain idea is my proposal, not a reward mechanism demonstrated or promised by Petals.

At the time of the video I describe BLOOM and LLaMA support and show a short Python example. It imports a tokenizer and Petals distributed model class, loads a supported model, and generates text. The example illustrates inference; current model support and installation details may differ.

I plan to try Petals as both a client and a server and invite interest in a longer tutorial. I am excited by the possibility of sharing idle GPU resources, while the video itself does not demonstrate my completed tests or establish the privacy properties of all participating hosts.