Using OpenLLM to Serve Open Models Through an API

In this June 2023 tutorial I introduce OpenLLM, an open-source project intended to make serving and building applications with open models easier. It exposes a simple API, supports several model families and offers integrations and deployment options. The commands and feature status in this walkthrough reflect the version shown at the time and may have changed.

The repository lists supported families including StableLM, Falcon, Dolly, FLAN, ChatGLM and StarCoder. It describes API testing, integrations with the BentoML and Hugging Face ecosystem, and ways to package a server or deploy it through BentoCloud. Fine-tuning is described as coming soon, not a feature I demonstrate.

I first create a separate Conda environment named `ol` with Python 3.11.3, then activate it and install OpenLLM using `pip install openllm`. An isolated environment helps avoid dependency conflicts. I run `openllm -h` to confirm the command is available and inspect its options.

I start the default OPT example with `openllm start opt`. Once the local server is running, I open its documentation interface at `localhost:3000`. The interface exposes generation and metadata endpoints, and it also lets me send a request without writing a separate client.

To test generation, I choose the endpoint, enter a prompt asking for a joke in the JSON payload and execute it. The page displays response codes and generation settings such as maximum new tokens, minimum length and early stopping. Server logs show the requests arriving.

The generated response is not particularly coherent, which I attribute to the small model used in this example. A successful API call shows that the serving path works; it does not establish that the model's answer is accurate or useful.

Next I write a short Python script that imports `openllm`, creates an HTTP client for the local server, sends a query and prints the response. The demonstration produces BentoML warnings about JSON encoders, but the request returns. Its explanation of “further” and “farther” is again nonsensical, illustrating the difference between connectivity and output quality.

The version shown supports additional models through optional packages. I open a second terminal, activate the same environment and install the Falcon extra with `pip install openllm[falcon]`. I note that my first attempt had missing dependencies; OpenLLM reported them and I installed them separately before retrying.

I start a Falcon server with `openllm start falcon`. The console mentions dependencies such as einops, xformers and safetensors, and the metadata endpoint identifies the model as Falcon 7B. The exact dependencies and installation process depend on the model and software version.

For deployment, I demonstrate `openllm build falcon`, which creates a Bento package for the model. I also explain that a model ID can select a specific variant when starting the server. The tutorial shows a packaging command, not a complete production deployment.

Finally, I point to Hugging Face agents integration and discuss LangChain support. The video describes some integrations and fine-tuning as upcoming, so consult the project's current documentation before assuming that a 2023 feature status or command still applies.