BloombergGPT: Mixing Financial and General Data for a Language Model

In April 2023 I review BloombergGPT, a 50-billion-parameter language model developed for finance. I am excited by the idea of combining a company’s specialized financial archive with general-purpose text, rather than training only on one or the other. Bloomberg published a research paper describing the model; that announcement alone did not mean its weights were publicly released.

The [paper](https://arxiv.org/abs/2303.17564v1) describes approximately 363 billion tokens of financial documents and 345 billion tokens from public general datasets. The financial collection, FinPile, includes English-language financial web pages, news, filings, press releases and Bloomberg archive material. These are corpus sizes, not a claim that every token is proprietary or that all data was consumed once during training.

Bloomberg augments FinPile with public sources including The Pile, C4 and Wikipedia. Components shown include GitHub, Books3, FreeLaw, Stack Exchange, patent text, YouTube subtitles, Hacker News and Enron emails. Their inclusion in the table is not a claim about the quality or rights status of every item.

I cite a 54.2% figure for financial data from a heading in the paper. Its Table 1, however, lists FinPile as 51.27% of the 708.9-billion-token corpus and public data as 48.73%; the two figures in the paper are inconsistent. The table lists financial web text as 42.01% and C4 as 19.48% of the full corpus, not of the public subset.

The model uses ALiBi positional encoding; the paper says it can be applied to sequences longer than its 2,048-token training length. That is an architectural possibility, not a demonstrated guarantee of equally reliable performance at arbitrary context lengths.

Training used Amazon SageMaker instances with eight NVIDIA A100 40 GB GPUs per instance. I point out that such hardware is expensive and that training a model of this size requires substantial resources. My observations about GPU prices are anecdotal and reflect 2023.

The researchers evaluate both finance-specific and general-purpose tasks, comparing BloombergGPT with GPT-NeoX, OPT and BLOOM. On a held-out FinPile language-modeling evaluation, the chart uses bits per byte, where lower is better. The researchers report that BloombergGPT performs better across the displayed categories; that is a specific evaluation, not proof it is best on every financial task.

The biggest gap in the chart is on filings. The authors suggest a possible reason: public filings often appear as PDFs and may not have been included in existing datasets. This is their explanation for the observed difference, not proof that other models were incapable of parsing all PDFs.

I consider whether a model trained on valuable financial data might eventually be a paid product, drawing on Bloomberg Terminal as an example. My suggested pricing and access are speculation; my remark that a Terminal costs millions of dollars a year is not a substantiated per-user price and should not be relied on.

I say the paper does not specify whether it will release the model, but its openness section actually states that the authors were not releasing it, citing concerns including extraction of FinPile data from model weights. They also discuss the value of open evaluation and potential misuse. My prediction about expensive access is separate from that stated decision.

I generally prefer organizations to release models openly, and I am excited to see new specialized systems. The research describes benchmark results and a training approach; it does not establish that the model is the world’s most valuable or offer financial advice.