Running a Local Language Model as an Offline Backup

Matthew Berman asks whether a local language model could make useful information available during a prolonged outage. He considers not only food and water but also access to knowledge about equipment, medicine and energy. In this January 2024 video, he walks through choosing and storing a model, running it locally and preparing backup power.

Berman describes language models as highly compressed knowledge and imagines them preserving humanity's knowledge after a disaster. That framing is an overstatement: model weights are not a complete or faithful archive, and a model can confidently produce false answers. They should not be the sole source for emergency, medical or other high-consequence decisions; keep independently verified offline references too.

He invokes the Library of Alexandria as an example of lost knowledge and contrasts it with the many physical and digital copies of information today. He is interested in the remaining points of failure, even though he describes the end-of-the-world scenario partly tongue-in-cheek.

A temporary internet outage can prevent access to cloud services such as ChatGPT and Anthropic's models. Berman suggests that a downloaded model running on a local computer can still answer questions without a connection. He also imagines permanent outages, restrictions on AI access, power loss and damage to electronics.

His basic setup has three parts: data storage, a computer capable of running inference, and a power source. He shows a small external solid-state drive containing a downloaded model, with room to spare. A working setup also depends on compatible software and hardware, and the files need to remain readable.

For model selection, Berman favors a recent model and the largest one the computer can actually run. He recommends quantization to reduce size and improve performance, with some potential quality loss. The best trade-off depends on the model, quantization method, available memory and the tasks being tested.

He also favors a model tuned to refuse fewer requests, arguing that an emergency might involve questions a more restrictive model declines. Less refusal does not make an answer accurate or safe, however, especially when a model is asked about dangerous technical work. Practical expertise and reliable references remain necessary.

For running the model, Berman recommends LM Studio as an accessible graphical application with model downloads and a chat interface. He also names Ollama as a command-line-oriented option and text-generation-webui as another interface. These are the tools and versions he discusses in 2024.

He recommends keeping both the software and model on the computer and on an external drive. He compares SSDs, magnetic tape and Blu-ray, describing SSDs as fast and optical storage as slower. His suggested lifetimes—about five years for an SSD, 10–15 for tape and 30–50 for Blu-ray—are rough claims, not guarantees; actual retention depends on media, handling and storage conditions.

For physical protection, he discusses a fire-resistant safe, a Faraday bag and desiccant. He says his safe is rated for a limited fire exposure and suggests a bag to reduce signal interference. Protection depends on the product's rating, the threat and how it is used; a bag or safe should not be assumed to protect against every signal, fire or other hazard.

He adds desiccant to reduce moisture, then demonstrates LM Studio. At the time he downloads the version for his operating system, places the Mac application on an external drive and opens it. The interface he shows supports model search, downloads and local chat; the exact layout may differ in other versions.

In the model search he chooses Dolphin 2.7 Mixtral 8x7B, a fine-tune by Eric Hartford, and points to LM Studio's compatibility estimate. He selects a Q5_K_M quantized file of roughly 32 GB. The on-screen 'should work' indicator is a heuristic, so the practical test is to load and run the chosen model on the actual machine.

After downloading, he checks the local models folder in LM Studio and changes it to the external drive if necessary. He then opens the chat tab, enables Apple Metal on his Mac and loads a model for a test. The model shown in the chat is a Laser Dolphin Mixtral variant; this demonstrates loading a local file, not a comparison of the two variants.

His test asks how to build a fire with sticks, and the model produces a list of steps. Berman notes that having a written answer is different from having the skill to carry it out. A plausible-looking response is also not proof that the instructions are complete or correct.

For loss of power, he considers a battery backup, rooftop solar or a fuel-powered generator. As an example, he describes a 1,200 Wh portable battery he estimates cost $600–$700 at the time, while noting that a smaller battery may suffice. Actual runtime depends on the computer's power draw and usable capacity.

Maintenance is part of the plan: Berman suggests periodically downloading newer models, updating the software and checking battery health. He notes the tension between keeping a battery ready and the wear associated with leaving it fully charged. His final claim that a tiny drive stores all human knowledge should be read as enthusiasm for the idea, not a literal description of what a model preserves.