PrivateGPT 2.0: A 2023 Guide to Local Document Chat and Its Roadmap

Matthew Berman introduces a new version of PrivateGPT, a project for asking questions about documents with a language model. The demonstration concerns the November 2023 release: the repository and installation workflow have since evolved, so readers should use the [current project documentation](https://docs.privategpt.dev/introduction) for a present-day setup.

The attraction is the possibility of keeping document processing on your own computer. Berman contrasts the original, simpler release with a more modular version that provides an API and allows components to be changed. Local operation is a configuration, however: choosing a remote model, embedding service or database can send information off the machine.

The video is sponsored by ServiceNow, whose Now Assist product Berman introduces for enterprise workflows. The sponsorship is separate from the PrivateGPT installation and demonstration that follow.

Berman opens the project repository and clones it to his machine. His [published 2023 installation notes](https://gist.github.com/mberman84/9b3c281ae5e3e92b7e946f6a09787cde) record the commands shown here; they are useful as a record of the video, not a substitute for the current documentation.

For the demonstration he enters the repository directory, creates a Conda environment with Python 3.11 and activates it. He then uses Poetry to install the UI and local dependency groups, with `poetry install --with ui,local`. Berman walks through the installation rather than assuming the viewer already has the components in place.

The project configuration separates choices such as the language model, embeddings and vector store. Berman examines the settings before running the setup script. The base settings and the selected profile both matter: a value visible in a base configuration file is not necessarily the effective value when a local profile overrides it.

The historical setup command, `poetry run python scripts/setup`, downloads the model and embedding assets. The demonstration uses a GGUF version of Mistral 7B Instruct, a BGE small embedding model and Chroma for local vector storage. Downloading the assets requires an internet connection even when later inference and retrieval are run locally.

Berman discusses Windows and Mac installation and mentions a historical llama.cpp Metal build command for Apple hardware. Hardware acceleration and package compatibility depend on the version and platform; check the applicable installation instructions and investigate errors rather than assuming build warnings or failures are harmless.

He starts the 2023 local profile with `PGPT_PROFILES=local make run` and opens the local web interface on port 8001. The demonstration provides separate modes for chatting with documents, chatting with the model and examining relevant context. These historical commands and interface details may not apply to the current release.

To test document chat, Berman uploads a paper about AutoGen and waits for ingestion. The system processes the document into chunks, creates embeddings and stores them so that relevant passages can be retrieved when he asks a question. He asks for a summary and observes the response generated from the material.

The context mode makes the retrieval step visible: instead of seeing only the final answer, the user can inspect the passages selected from the document. This is useful for checking whether the system has retrieved material that actually supports a response; retrieval and generation can still make mistakes.

He also uploads a Harry Potter PDF and asks questions about it. A fluent or correct answer to a familiar book is not, on its own, proof that the answer came from the uploaded file, because the model might already have encountered the information. Inspect the returned context and compare it with the source.

Beyond the web interface, Berman opens the API documentation and points to endpoints for ingesting documents, listing them and requesting completions. The developer interface is intended to let applications use the document pipeline without depending on the demo UI.

Berman introduces Iván Martínez, the creator of PrivateGPT, for a short conversation about the project. Martínez describes seeing organizations hesitate to send their information to external AI services. Running open models and retrieval tools on hardware under their control was a way to make experimentation more approachable.

Martínez says the local configuration can run without an internet connection once the necessary dependencies and models are available. That is a useful privacy property, not an automatic guarantee: users still need to check the actual model and embedding providers, stored files and logs, access controls and network exposure.

Looking ahead from 2023, Martínez discusses adding connections to additional data sources and making workflows such as summarization and extraction easier. He also mentions observability and evaluation so developers can understand and assess how their applications perform. These are plans described in the interview, not a claim that each feature shipped as described.

He also envisions deployment on a cloud instance and more distributed arrangements, including managed model services such as SageMaker and external vector databases. Those options can help scale an application but change the privacy boundary: a remote component can receive data. Berman closes by thanking Martínez and directing viewers to the project.