Testing Local AI on a Dual RTX 6000 Workstation

요약

30초 요약

In this June 2024 video, Matthew Berman tries local models on a Dell Precision 5860 workstation with two NVIDIA RTX 6000 Ada Generation GPUs. He reports speeds for several quantized and full-precision models, loads multiple instances in LM Studio, and tries both code and image generation. Dell and NVIDIA supplied the workstation, and the video description discloses Berman’s investment in LM Studio. The speeds and resource readings are informal observations from this setup, not controlled benchmarks; the 70B model’s generated Snake game did not work in his test.

1Workstation configurationThe supplied Precision 5860 has two RTX 6000 Ada Generation GPUs with 48 GB each, plus 128 GB of system RAM.
2Precision changes memory needsBerman says a 70B model at 32-bit precision would require about 280 GB for weights; his Q8 version fits within the machine’s GPU memory.
3Small-model speedA Llama 3 8B Q4 story generation runs at about 60 tokens per second in his test.
4Ten concurrent instancesTen Q4 instances run together at roughly 11 tokens per second each; the software then declines additional loads.
5A 70B model locallyThe Llama 3 70B Q8 example generates at about nine tokens per second on this setup.
6Generated game failsThe model’s Snake program contains problems and still does not run after Berman’s edits.
7Image generationA Fooocus demonstration yields dog images, which Berman estimates took around four seconds apiece.

핵심 정리

Two 48 GB GPUs provide 96 GB of aggregate VRAM, but model precision, overhead, and the software setup determine what fits.
Berman reports roughly 60 tokens per second for a quantized Llama 3 8B run and about nine for the larger Llama 3 70B Q8 run.
Concurrent model instances increase the load: ten Q4 instances run at about 11 tokens per second each in his demonstration.
A model loading limit and GPU memory usage are distinct; the video does not establish why nine Q8 instances were the maximum in that attempt.
Fast inference does not ensure correct output: the tested 70B model’s Snake program failed even after edits.
His local image-generation demonstration produced images at an estimated four seconds each under that particular setup.

주제Local inference Inference 모델과 학습 Generative media 개발 워크플로와 테스트

전문

챕터별로 다듬은 전문 읽기

원본 자막과 대조

원본 자막만 보기

The supplied workstation

Berman introduces a Dell Precision 5860 sent by Dell and NVIDIA and lists its components. He also describes whom he thinks such an expensive system could serve.

I start by running ten instances of Llama 3 8B Instruct at once. Dell and NVIDIA sent me a workstation that I want to push with large local models and concurrent inference. The video description also discloses that I am an investor in LM Studio, the software I use for the language-model tests.

The machine is a Dell Precision 5860 Tower with an Intel Xeon W-2475X processor and Windows 11. It has two NVIDIA RTX 6000 Ada Generation GPUs, each with 48 GB of GDDR6 VRAM, for 96 GB in aggregate. I also list 128 GB of DDR5-4800 system RAM and two 4 TB M.2 SSDs.

The machine is expensive and not for everyone, but I think its large GPU memory could be useful for people running large local models, fine-tuning, or doing data science. It is also smaller than my previous tower. Those are my impressions; this video tests inference rather than demonstrating a training or fine-tuning benchmark.

Choosing models and precision

He opens LM Studio and inventories models with different sizes and quantization levels. He discusses the memory trade-off for a 70B model.

I install LM Studio and download several models, using about 224 GB of disk space. The largest is a Q8 version of Llama 3 70B Instruct at just under 75 GB. Quantization matters because the space required depends not only on the number of parameters but also on the numerical precision used for the weights.

I estimate that 70 billion parameters at 32-bit precision would take roughly 280 GB just for the weights, far beyond the combined 96 GB of GPU memory here. The Q8 model is much smaller. Actual memory requirements also depend on format, runtime overhead, context, and how the workload is distributed across devices.

My other downloads include Mistral 7B Instruct v0.3 in FP16, Phi-3 Mini 4K Instruct in FP16, Llama 3 8B Instruct in Q8 and Q4, and Phi-3 Medium in a full-precision format. I plan to try small models first, then larger ones and finally local image generation.

Single-model inference

Berman runs small Llama, Phi, and Mistral models and reports observed generation speeds. These are demonstrations on a particular system, not a controlled comparison.

I load Llama 3 8B Instruct in Q4 with full GPU offload and ask for a story. The interface shows about 60 tokens per second. I also ask it to write Snake in Python and observe a similar generation speed. These numbers describe individual runs rather than a standardized benchmark.

슬라이드 1 / 5Llama 3 8B generation speedThe interface shows approximately 60 tokens per second in one single-model run.

Next, Phi-3 Mini 4K Instruct in FP16 generates a story at about 63 tokens per second. I also give it a puzzle involving killers in a room. Its response is confusing, and I am unsure whether it is correct; the speed measurement is separate from the quality of the answer.

I continue with a Q8 Llama 3 model and report about 56 tokens per second, then load Mistral 7B Instruct v0.3 in FP16. The Mistral run produces a long story at roughly 42 tokens per second while I observe around 40% GPU utilization. Model size, settings, and prompt affect these informal results.

Concurrent instances

He loads multiple Llama 3 8B instances and watches speed, utilization, and memory. The interface limits further loads, with the cause not established.

I try generating stories from several model instances in parallel. With the earlier group I see roughly 25 tokens per second per instance. I then load ten separate Llama 3 8B Instruct Q4 instances and prompt them together; the displayed rate falls to about 11 tokens per second per instance.

슬라이드 2 / 5Concurrent Llama 3 instancesLM Studio displays several Q4 model instances generating responses simultaneously.

While ten instances are running, I see GPU utilization fluctuating around 65–80%, a temperature near 150°F, and relatively low CPU and system-memory use. The fans become a little more audible. When I try to load five more models, the interface says to eject some first, even though it reports roughly 56.84 GB of the 96 GB VRAM in use.

I eject the Q4 models and try Q8 instances, which I estimate need about 8.5 GB each. Only nine load in this attempt. I guess that the software does not allow all the memory to be filled, but the video does not establish the actual reason; the reported rate is still around 11 tokens per second.

슬라이드 3 / 5Nine Q8 model instancesThe loaded-model list illustrates the separate Q8 instances used in the concurrency test.

Larger models and a coding test

Phi-3 medium and Llama 3 70B run locally at different observed speeds. The 70B model produces a Snake program that fails to run after attempted fixes.

I switch to a larger Phi-3 Medium model in full precision. It produces a short story at about 27 tokens per second, faster than my initial guess. I then load the Q8 Llama 3 70B model: its story runs at approximately nine tokens per second with around 75 GB loaded into GPU memory.

To try a practical coding prompt, I ask the 70B model to write Snake in Python using Pygame. The response finishes at around nine tokens per second. In VS Code I notice undefined width and height references, ask the model for a correction, and make further edits using instance attributes.

슬라이드 4 / 5Snake coding test in VS CodeThe generated program is examined and run after edits; the video reports that it did not work.

Even after those changes, the game does not work when I run it. The result shows that loading a large model and generating code quickly does not guarantee a working program in a single attempt.

Local image generation

Berman uses Pinokio and Fooocus to generate images locally. He gives an approximate per-image timing and closes the test.

For the final test, I use Pinokio to install Fooocus and run a local diffusion model. With a simple dog prompt, two images appear; I estimate about four seconds per image and like the results. That estimate applies to this particular model, settings, and run rather than all image-generation workloads.

슬라이드 5 / 5Local image generation in FooocusThe diffusion interface shows dog images generated from a short prompt.

I plan to keep testing the workstation, which has become my main machine, and invite suggestions for further workloads. The video closes with a hardware giveaway announcement.