Pascal Pfeiffer on Retrieval, Open Models, Kaggle and Fine-Tuning with H2O LLM Studio

Matthew Berman welcomes Pascal Pfeiffer, then a principal data scientist at H2O.ai and a Kaggle competitions Grandmaster. They discuss open models, retrieval, coding assistants, competitions and a demonstration of H2O LLM Studio in this June 2023 conversation.

Pfeiffer says H2O.ai began with open-source statistics and data-science tools before expanding into automated machine learning. Its products included open tools and enterprise offerings such as Driverless AI, which automates parts of model building and comparison.

He credits the company’s collection of Kaggle Grandmasters and other specialists with bringing practical competition experience into products. Berman asks whether H2O is mainly business-to-business; Pfeiffer distinguishes enterprise AI Cloud from open tools intended for a broader audience.

Pfeiffer says the released fine-tuned Falcon models and related tools can be used on suitable local or cloud GPUs. Hardware requirements vary by model size and configuration; a single-GPU deployment does not imply every model fits every consumer GPU.

Berman asks why H2O’s Falcon service felt faster than his earlier attempt. Pfeiffer says autoregressive generation produces tokens sequentially, then credits Hugging Face Text Generation Inference and FlashAttention with accelerating their deployment rather than taking credit for the underlying optimization.

Pfeiffer describes FlashAttention as an efficient implementation of attention that combines operations to reduce overhead. He says tensor parallelism can distribute matrix operations across GPUs but was not used in the single-GPU deployment Berman tried.

Asked about local use, Pfeiffer says H2O publishes model weights and model cards on Hugging Face. Users can consult the code examples and choose compatible serving software; the exact requirements depend on each model and its license.

Berman asks how the H2O chat interface answers questions about uploaded documents. Pfeiffer describes a workflow using tools such as LangChain to connect document processing, retrieval and the language model.

A PDF is converted to text and divided into chunks because a long document may exceed a model’s context window. Searching for a specific answer by passing every chunk through the model would add cost and latency and can still produce incorrect answers.

Pfeiffer explains embedding document chunks into a vector space and comparing them with an embedding of the user’s query. The system retrieves relevant text and supplies it alongside the question to the language model; retrieval is a way to ground an answer, not a guarantee against hallucination.

Berman restates the pipeline: chunk and index the document, retrieve relevant pieces for a query, then pass those pieces and the question to the model. Pfeiffer notes that multiple chunks can be included and that more elaborate summarization or per-chunk processing adds computation.

Berman raises a practical limitation of early coding agents: they could create a project from a prompt but struggled to iterate on a large existing repository. He asks whether a vector database could give the model enough context to modify interconnected code reliably.

Pfeiffer calls the problem difficult and says repository-wide understanding remained unsolved in 2023. Semantic retrieval can find relevant files, but dependencies may span more code than fits in a prompt, and real changes often require repeated testing and debugging.

He considers larger context windows, multiple candidate reasoning paths and sandboxed execution as possible approaches. These are ideas and predictions, not evidence that one technique solves the problem for every codebase.

Berman proposes generating a map of the repository: file summaries, function signatures, parameters and expected outputs could bridge a query and the retrieved code. He asks whether such a map would help select the right context for an edit.

Pfeiffer compares the idea with headers and documentation. A concise map could connect pieces of a clean codebase, but real projects are messier and automatically generated docstrings can be wrong. Both regard it as an idea worth testing rather than a finished solution.

Berman says persistent context would make a coding assistant more useful for iterative work. Pfeiffer imagines a junior pair programmer that could help developers, while acknowledging how difficult it is to build one reliably.

The conversation turns to open and proprietary models in 2023. Berman asks whether smaller open models truly reason as well as closed ones or mostly imitate their answers. Pfeiffer expects broad community participation to help open models over time, while describing GPT-4 as a strong general-purpose model at the time.

Pfeiffer says smaller models can handle many ordinary non-coding questions, but notes that some are trained on outputs resembling those of stronger models. Imitation of responses can be useful without proving equal capabilities across all tasks.

He cautions against headline claims that a model reaches a fixed percentage of GPT-4 or GPT-3.5 performance. The answer depends on the questions, evaluation procedure, coding checks and potential biases in human or automated judgments.

Berman asks whether a central model could route requests to specialist models, as in the HuggingGPT idea. Pfeiffer says specialization could help, but routing unexpected requests requires a strong classifier and introduces its own complexity.

They discuss a contemporary rumor about GPT-4’s internal architecture. Neither has first-hand confirmation, and the proposed number and size of component models remain unverified; they use the rumor only as a starting point for discussing possible designs.

Pfeiffer explains one possible ensemble: train multiple versions of a model and combine predictions, such as by averaging or voting where appropriate. Different runs can make different errors, so a combination may improve average performance, but it also increases inference cost.

He says such blending is common in Kaggle when competitors seek the last increment of accuracy. Production systems must weigh that improvement against serving multiple models, which can multiply computational expense.

Pfeiffer offers another possible architecture: a router sends requests to different specialized models. He emphasizes that he does not know which architecture, if any, GPT-4 actually used; the conversation remains hypothetical.

Kaggle is a platform for data-science competitions and knowledge sharing, Pfeiffer explains. Organizers provide data, a task and an evaluation metric, and participants develop models that are scored on public and hidden test sets.

The hidden evaluation set matters because participants cannot directly tune against the final answers. Berman offers a hypothetical company recommendation problem; Pfeiffer says competition data may be public, anonymized, transformed or synthetic depending on the organizer and task.

Pfeiffer found Kaggle near the end of his PhD while looking for practical machine-learning problems. Textbook examples had not shown him all the difficulties of real data, so he valued designing validation and a pipeline without seeing final test labels.

He describes a sharing culture in which participants discuss discoveries and publish notebooks. Reading others’ code and working on the same dataset accelerated his learning and encouraged him to continue competing.

In a typical image-classification competition, entrants receive labeled training images, a target metric and a test-data format. They build a training and inference pipeline, potentially starting with a pretrained image model and adding a classifier.

Some competitions require code submissions that run predictions against unseen test data and produce a submission file. Limits on daily submissions constrain repeated probing of the public leaderboard, although validation and overfitting remain important concerns.

Leaderboards provide feedback and make the process engaging. Pfeiffer says prizes can vary widely: some competitions have unusually large awards, while many distribute smaller pools among a limited number of winning teams.

What makes a solution best depends on the organizer’s metric, such as F1 for a classification problem. Pfeiffer recalls that badly chosen metrics can prompt community discussion and changes, and warns that a convenient score is not always aligned with the actual business goal.

Asked about Grandmaster requirements, Pfeiffer recalls five gold medals including one solo result, while expressing uncertainty about the exact medal allocation rules. Those rules are specific to the competitions track and can change; he does not present his recollection as definitive.

For beginners, he recommends joining a competition, reading shared notebooks line by line and following discussions rather than simply copying code. The aim is to identify useful signals, understand potential bias and keep learning as techniques change.

Berman turns to H2O LLM Studio and the frequent request to fine-tune on private data. He suggests prompting and retrieval as alternatives for many applications; his estimate that they suffice for 99% is an opinion, not an established statistic.

Pfeiffer opens the 2023 LLM Studio interface and explains that it supports fine-tuning a language model on a user dataset. Chat is a common use case, but examples can also train models to produce structured outputs or follow API-related formats.

He describes fine-tuning largely as teaching output style and behavior on top of a pretrained backbone. That is a useful description of the showcased instruction tuning, though fine-tuning can also alter knowledge and other capabilities depending on the data and method.

The demonstration uses the OpenAssistant dataset, with instruction and response examples. Pfeiffer says other tasks can use different mappings, such as a passage as input and a JSON-like record as output; multi-turn conversations can use parent identifiers.

He creates an experiment by selecting a dataset, causal language-modeling task and model backbone. A decoder-only language model predicts the next token from preceding tokens, and the chosen pretrained backbone supplies the starting parameters.

Pfeiffer shows personalization settings that can replace a bot’s name or author in training examples. He then discusses special tokens marking the start of a user prompt, the assistant answer and the end of a response.

End-of-sequence markers can teach a model where an answer should stop; without an appropriate stopping convention, a base model may continue generating. He also points to tokenizer settings and limits for input and output lengths.

The interface offers several numerical precisions and quantization options. Pfeiffer explains that lower precision can reduce memory requirements, with a possible quality tradeoff. Whether a particular model fits an 8 GB GPU depends on model size, context, method and other overhead.

He explains LoRA, low-rank adaptation: the pretrained weight matrix is frozen and a small trainable update is represented by two lower-rank matrices. This reduces the number of trainable parameters and associated optimizer state relative to full fine-tuning.

Pfeiffer illustrates how a large weight matrix can be approximated by factors with a much smaller intermediate rank. Combining LoRA with quantization can lower memory requirements, but model size and training configuration still determine the hardware needed.

LLM Studio offers evaluation options including perplexity, BLEU and model-based scoring, and supports distributed data parallel training across multiple GPUs. Each metric measures a different aspect of performance; a lower training loss alone does not establish reliable real-world behavior.

For the shown setup, Pfeiffer says a Falcon 7B experiment took about three hours on three GPUs, while a 2.8B run was projected to finish in about 30 minutes. These timings are examples from his hardware and settings, not general performance guarantees.

He shows the training loss and logs and says the project is open source and configurable. Users can inspect the repository and report issues; training data, evaluation and deployment still require their own careful decisions.

Pfeiffer attempts to open the chat interface for an earlier trained model, but the development version produces an error. He explains that the intended workflow is to chat with checkpoints and compare validation metrics; the live demo does not successfully complete that chat.

He adds that LLM Studio also has a command-line pipeline, not only a graphical interface. For installation and experimentation he points viewers to the H2O LLM Studio GitHub repository and the hosted H2O GPT interface.

Berman thanks Pfeiffer for the discussion of Kaggle, open models, architecture and fine-tuning. Pfeiffer shares where viewers can find him and closes the interview.