Matthew Berman on Using Idle Computers for AI Inference

SUMMARY

In 30 seconds

Matthew Berman describes Darkbloom, a project that connects idle personal computers to a distributed system for running open-source AI models. He suggests that such an approach might help address concerns about expanding data centers and points to publicly available code as something users can inspect. The video presents an optimistic possibility, not evidence that distributed inference can replace data centers or eliminate their energy use. The project's own materials describe a more specific system and hardware requirements.

1Data center expansionBerman describes the rapid construction of data centers for AI and says many people dislike the expansion.
2Put spare compute to workHe introduces Darkbloom as a way to connect idle computers to a distributed system that powers open-source models.
3Inspectable codeBerman says users concerned about installation can audit the public code. Public access is useful for inspection, but it is not a security guarantee.

Takeaways

Berman sees growing AI data centers and public opposition to them as motivation for finding alternative computing capacity.
Darkbloom proposes using idle compatible computers to serve AI inference requests across a distributed network.
Publicly available code can be examined, but openness alone does not establish security or privacy.
The video does not establish that the approach can replace data centers or reduce total energy use.

TopicsDistributed training & inference Infrastructure & deployment Inference Open-weight models Energy

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The data center problem

Berman connects the growth of AI to the expansion of large data centers and public dislike of that buildout. He suggests there may be another source of compute.

Nobody likes data centers right now, and this might be a solution. Much of what you do on the internet involves large data centers. With the rise of AI, they are being built out on an enormous scale, and a lot of people dislike that expansion.

Idle computers and open code

He describes Darkbloom as a distributed system that can use idle personal computers to run open-source models. He notes its publicly available code and expresses optimism about the project.

There is a project called Darkbloom that I think might be an answer. It lets you connect a computer to a distributed system of compute. When your computer is sitting idle, its spare capacity can be used to power open-source models.

The codebase is publicly available, so if you are concerned about installing it, you can inspect the code yourself. People are starting to use it and get excited about it. Editor's context: [Darkbloom](https://www.darkbloom.ai/) describes a service using Apple Silicon Macs for decentralized inference, so the video's reference to idle computers should not be read as support for every machine. Its [public provider repository](https://github.com/Layr-Labs/d-inference) describes separate coordination infrastructure; inspectable provider code does not by itself establish that the whole service is open or secure. The video gives no measurements showing that this approach replaces data centers or reduces overall electricity use; hardware availability, network performance, privacy and energy consumption remain relevant considerations.