Installing LLaVA on a GPU Server: Matthew Berman’s 2023 Walkthrough

Matthew Berman introduces LLaVA, a visual language model that can answer questions about images. In this October 2023 demonstration, he presents it as an alternative to ChatGPT Vision and walks through the setup he used on a rented GPU.

The project builds on the Visual Instruction Tuning paper and an open-source repository. Berman says the version he is demonstrating uses a fine-tuned Llama 2 13B model. He shows a cell diagram and says it can identify and explain the labeled parts.

Berman says he encountered Linux-specific dependencies on Windows and CUDA requirements on his Mac, so he chose a Linux GPU instance on RunPod. He suggests that a suitable Linux computer could run the software locally and that Windows Subsystem for Linux might work, though he does not demonstrate it.

In the 2023 recording, he chooses an RTX A6000 instance advertised at $0.79 per hour, noting that a smaller GPU might suffice. He changes temporary container storage from 20 GB to 100 GB for model downloads and exposes port 3000. The cloud instance is paid even though the project code is free.

After deployment, he opens a web terminal, clones the LLaVA repository and enters its directory. He upgrades pip, installs PyTorch and installs the project requirements. He refers viewers to the instructions linked in the video description for the complete commands.

He starts the controller on port 10000, then opens a second terminal and starts the model worker. The worker downloads the model on its first run; he estimates that the model needs more than 20 GB of storage. Both processes need to remain running.

During his download, the terminal shows repeated error-level lines alongside progress. Berman says he was able to ignore those particular messages, though he does not establish that every error can be ignored. The worker then loads checkpoint shards and starts serving the model.

In a third terminal, he configures the Gradio server to use port 3000 and starts the web server with the controller address. Opening the exposed port in RunPod brings up the LLaVA interface, including a model selector and image upload.

The first example asks what is unusual about a picture. The model describes a person ironing while standing on the back of a moving car. Berman treats this as a demonstration of image understanding, not a systematic evaluation.

He reiterates that the RunPod machine costs money, while a suitable local Linux setup could avoid that rental. He says he could not get a Mac fork or the Apple silicon instructions working in his own attempt.

For another test, Berman uploads a labeled cell illustration and asks the model to explain its terms. It begins describing the cell and defining components, even though the image only contains labels. He notices a failure: the response enters a loop and repeats the list.

The video does not explore the full limits of the model. Berman points viewers who do not want to install it toward a linked web demo. After recording, a contributor also created a one-click RunPod template, linked in the description.