Running LLaMA and Alpaca Locally: A 2023 Dalai Tutorial

RÉSUMÉ

En 30 secondes

In this 2023 tutorial, Matthew Berman demonstrates the Dalai tool for running early LLaMA and Stanford Alpaca models locally. He covers installation, starting a local server and changing generation settings. The commands and model availability are historical and may no longer work as shown. Stanford's reported sub-$600 cost covered generating instruction data and fine-tuning an existing LLaMA 7B model, not pretraining it from scratch. Its preliminary comparison with text-davinci-003 was limited in scope and did not establish parity with ChatGPT or GPT-4.

1A small instruction-tuned modelBerman introduces Stanford Alpaca as a derivative of Meta's LLaMA 7B. Stanford's reported cost covers data generation and fine-tuning, not the original pretraining.
2Historical local setupHe installs Dalai and model files, then starts a local server. Tools, downloads and license terms may differ today.
3Output lengthA response stops at the n_predict limit, so Berman raises it to allow a longer answer.
4Cost comparison needs contextBerman compares Alpaca's reported cost with GPT-3 training. The numbers cover different stages and do not establish equal capability or equivalent total training cost.

À retenir

Alpaca 7B was fine-tuned from LLaMA 7B using 52,000 instruction examples generated with text-davinci-003.
The reported sub-$600 reproduction cost excluded the cost of pretraining LLaMA and should not be compared as a like-for-like full training cost.
The video demonstrates a historical Dalai setup with Node.js, model downloads and a local server on a MacBook Air.
The n_predict setting limits output tokens; increasing it can prevent a response from stopping at that limit.
The original LLaMA and Alpaca releases carried non-commercial research restrictions and the setup is a historical example.

ThèmesLocal inference Modèles et entraînement Fine-tuning AI cost & performance

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

Alpaca and its training recipe

Berman introduces Stanford's instruction-tuned model and its relation to LLaMA. The reported cost and model comparison require the scope of Stanford's preliminary experiment.

Stanford's Alpaca is an instruction-tuned model based on Meta's LLaMA 7B. In this 2023 demonstration, I run LLaMA and Alpaca locally on a MacBook Air using a tool called Dalai. The original models had research and non-commercial restrictions, and today's installation experience may be different.

Stanford reported that Alpaca behaved qualitatively similarly to text-davinci-003 in a preliminary single-turn instruction evaluation. Berman is impressed by the model's size and cost, while saying his own tests did not put it on par with GPT-4. That limited comparison does not establish broad equivalence to ChatGPT or GPT-4.

The often-cited cost below $600 is not the price of training the whole model from scratch. Stanford reported spending less than $500 on API-generated examples and under $100 on the initial fine-tuning run, using an already pretrained LLaMA 7B model. Its base-model pretraining cost is not included.

Stanford started with 175 human-written instruction-output pairs from the Self-Instruct seed set and used text-davinci-003 to generate roughly 52,000 examples. Those examples were used to fine-tune LLaMA 7B. Berman points viewers to the Stanford announcement and notes that its public demo was no longer available when he recorded the video.

Installing and running Dalai

He demonstrates a historical Node.js and Dalai installation, model downloads and a local server. The software and availability may have changed.

The tutorial uses the Dalai repository by cocktailpeanut. Berman says Node.js 18 or later is needed and demonstrates installing the package, then downloading LLaMA 7B and 13B and Alpaca 7B. He already has the models and cancels repeated downloads; one need not install both LLaMA sizes to follow the example.

DIAPOSITIVE 1 / 3Dalai commands in the historical tutorialThe video shows the Alpaca installation and local server commands used in 2023.

Once the model files are in place, Berman starts the local server with `npx dalai serve` and opens its local interface. He selects a model and asks how llamas and alpacas are related. These are the commands and interfaces shown in the 2023 video, not a guarantee of a currently supported setup.

Generation settings and cost claims

Berman adjusts the output length, describes other settings and discusses the falling cost of models. His cost comparison omits the base model's pretraining.

DIAPOSITIVE 2 / 3Output token limitThe n_predict field limits the generated response length in the demonstrated interface.

The answer stops when it reaches the `n_predict` limit of 200 tokens. Increasing the setting allows a longer response. The interface also offers a model selector, an optional remote server URL, a thread count, a seed and a temperature setting that affects variation in the output.

DIAPOSITIVE 3 / 3Dalai generation optionsThe repository documentation describes model selection, threads and generation settings.

Berman cites a prediction that AI training costs would fall rapidly and compares the cost of GPT-3 with Alpaca's reported cost. Those figures are not directly comparable: Alpaca's number covers data generation and fine-tuning on an existing pretrained model, and the preliminary evaluation does not prove equivalent performance. He closes excited about trying more models locally and comparing their strengths.