Chris Vela on Benchmarking vLLM Configurations with Runpod BenchPod

I’m Chris Vela, a forward deployed engineer at Runpod. I want to talk about benchmarking vLLM and the difficulties teams encounter when a new model appears. Runpod makes GPUs available from research and development through production serving, and I often see customers unsure how to launch a model or which settings to adjust.

Many teams start with a base configuration or a vLLM recipe. That is a useful starting point, but it does not answer whether the setup is right for production. Instead of guessing, I want data that can show an engineer or manager that a change actually works better.

Start by asking what the model is for: a coding agent, for example, has different needs from a simple summarization task. A very large context window is not always necessary. Expected user concurrency also matters; optimizing only for the smallest, cheapest GPU can miss how the KV cache will be used under load.

There are many configuration knobs to consider before production. Runpod pods are like virtual machines, use Docker and accept your own image. But even with convenient infrastructure, you still have to choose a compatible serving stack and configuration.

A newly released model may require a newer vLLM version than the one in a familiar recipe. When the underlying implementation is moving quickly, teams can encounter confusing errors or immature support. I would not necessarily put a model into production the day after release; the implementation may still need work.

To make experiments easier, I created BenchPod, a template that had not yet been released at the time of the talk. It is built around vLLM and centralizes serving configuration, metrics and benchmarking, so teams can change a configuration and compare results quickly.

One pain point is the restart and cold-start wait after changing a configuration. Downloading model weights and loading them into GPUs can take around 30 minutes in my example. Being able to test on actual GPU machines, including B300s when available, lets you check claims against measurements rather than relying on descriptions.

BenchPod uses a vLLM OpenAI-compatible Docker image and a FastAPI application. You can specify which models to serve, and a watchdog can restart the service when it fails. I describe an internal instance with four nodes where a model outage self-healed without disrupting our users.

Our team has been using Kimi K3 for coding agents. Although the model supports a one-million-token context, I say a single B300 cannot reach that full length in this setup and we use roughly half. In our experience that is still sufficient for long tasks and makes good use of the KV cache.

The dashboard includes GPU and host metrics, vLLM metrics, benchmarking, serving configuration, self-healing and a playground. CUDA compatibility matters too: I note that the demonstrated Kimi configuration works with CUDA 13.0 rather than 12.8. Teams can use the playground to test a model before connecting it to an agent.

The metrics also help teams observe cold starts and behavior under actual traffic. The template was not yet generally available during the talk. In the demonstration, I show the Runpod interface and note that available GPU supply can change; the screen happens to show no B300s available at that moment.

A pod can be deployed with a storage volume and the BenchPod image. The running instance exposes live metrics and error information, with data backed by storage and accessible through FastAPI calls. An agent can use the API without relying on the graphical interface, while the dashboard gives operators an overview.

I show a recent GPU failure and a roughly nine-minute cold start for the Kimi instance. The serving tab lets me stage an experiment using a vLLM YAML configuration and restart the service to apply it. For this demo, I varied the maximum number of batched tokens.

I ran the benchmarks in advance rather than waiting through them live. The runs use concurrency levels of 1, 8, 64 and 192 users, and the dashboard can compare multiple configurations. This makes it possible to see how a setting behaves as load increases rather than relying on a single test.

The displayed comparison illustrates a tradeoff. At concurrency 64, p99 inter-token latency falls from about 195 to 78 milliseconds with the larger batch-token setting, while median time to first token rises from about 1,966 to 2,348 milliseconds. The result depends on which latency matters to the application.

I observed that increasing the batch-token setting did not change first-token latency much in some cases, while inter-token latency dropped substantially in the example. For a long request, faster token generation can make the overall job finish sooner than with the base recipe; the comparison should be evaluated for the workload.

Once you have data from a pod, you can consider moving to a multi-node cluster or serverless infrastructure that scales up and down. Serverless users can also encounter delays while workers start, so measuring the actual behavior helps inform production decisions.

I demonstrate a command-line client using the running model. I say around 50 people are using our internal instance, and we also run other models. Looking ahead, I am interested in smarter routing so that teams can host and select among models rather than depend on one configuration.

You can design an experiment around the expected workload, such as hundreds of users. My recommended workflow is to pick a pod and GPU, select a model, compare configurations, and test with the agent. Optimize the metric that matters—such as inter-token latency, time to first token or time per output token—then ship the winner and scale it.

A frontier model is not automatically necessary for every team. I estimate that a B200 node on our platform costs around $40,000 a month, subject to price changes. It is powerful hardware, but the decision should reflect the actual task and cost.

For example, I run a DeepSeek model on four H200s for tasks it handles well, while I use Kimi for longer, more detailed work. Tooling and context management matter when self-hosting, and I see model routing and AI governance as next steps for Runpod.

I invite questions and note that I continue to learn new model-specific pitfalls as I test releases. The BenchPod template was planned for release at the end of the month so attendees could experiment with it themselves.