Polish Is 1% of Frontier Training Data, So Poland Built Its Own Model

Nathan Labenz frames Poland as a representative case, not a superpower. Thirty-eight million people and a GDP of roughly $1 trillion are about 10% and 3% of the United States. That is not enough money or compute to race the United States and China at the frontier, but it is enough technical talent, language pride, and political will to refuse being a client of whoever ships the next closed model. His guest is Marek Kozlowski, head of the AI lab at Poland's National Information Processing Institute and lead of project PLLuM — Polish Large Language Models.

Kozlowski pushes the idea slightly wider than "sovereign AI." He talks about localized LLMs: models adapted to a language *or* a domain, with better understanding and better generation in that slice. The quality target is blunt. Build models an order of magnitude smaller than the popular closed systems, and on language, culture, or domain, match the quality of models ten times larger. They should be open source, transparent, secure, and as organic as the team can make them. National models and bank-internal models, in this telling, are the same move at different zoom levels.

Labenz recalls that InstructGPT-era models, trained to follow English instructions, transferred into other languages almost by accident. Kozlowski's numbers are less comforting. In the biggest open and closed models, more than 90% of training data is English and Chinese; everything else shares 10% or less. Polish is about 1% in some corpora, and smaller in others. Most skills are therefore acquired in English and Chinese instructions. Transfer is real: a model taught math in English can, in intermediate steps, translate a Spanish prompt and still get the answer. What transfer does not buy is native texture.

His email example is small and lethal. A model that was not soaked in Polish will still produce something a reader can understand, then drop a formula that belongs to English: "I hope you stay in good health and condition." Word for word it is grammatical. No Pole would write it. Grammar, he argues, is not the language. Culture, habit, history, and which jokes are circulating this week sit in the gap between a correct sentence and a native one. Models can have a wide vocabulary and still fail to use an idiom in the right room, or know which places are historically loaded, or which ones are viral now.

Standard benchmarks will not catch that gap. Kozlowski is blunt about MMLU-style tests: they are mostly A/B/C/D questions. They measure understanding, extraction, summarization, and factual recall. They barely measure long-form generation, and almost never in Danish or Polish. "We live in a world biased by the benchmarks." His team built PLCC, the Polish Linguistic and Cultural Competency Benchmark, covering grammar and vocabulary but also culture, tradition, history, and the handling of ambiguous words. He admits the current version still does not score longer Polish structures. The thing they care about most is the thing the leaderboard still skips.

PLLuM did not start from random weights, for a boring reason: money and tokens. Open-model reports, Kozlowski says, show that even an 8-billion-parameter model wants at least a trillion tokens before pretraining is stable. After deduplication and filtering, his team had around 200 billion Polish tokens — an order of magnitude short. The method is language adaptation. Take Llama and Mistral bases (a year earlier their licenses were less hostile to the EU), continue pretraining for a few epochs on the Polish corpus, and only then run SFT and preference learning. English ability sags through catastrophic forgetting. They do not prune other languages by hand; they just train extra on Polish and accept the trade-off. Mixture-of-language experiments aimed at a from-scratch trillion-token run were still in flight when they spoke.

He maps the three stages onto childhood. Pretraining is learning words, structure, and scattered facts — a child after that still cannot do math or write an essay. SFT is school: poems, summaries, calculations, the actual competencies. Preference learning is the marked test, the signal of what to correct. If any of those stages is fed linguistically poor synthetic instructions from another LLM, the model degrades. So they invest in what he keeps calling organic data. Dozens to hundreds of annotators write instructions and preferences, or a machine drafts them and a human rewrites every line. The tools are in-house. There is no Scale AI equivalent and no crowdsourcing. Most of that data stays closed; samples go to Hugging Face so outsiders can see the shape.

Two weeks before the taping they put a nearly 100-page paper on arXiv, *The PLLuM Family*, the cookbook as well as the weights. Open source, Kozlowski says, is not only open weights. It is samples of instructions and preferences and a step-by-step recipe, which the most popular "open" models still do not publish because those traces are intellectual property. A quieter novelty: models hardened for English speakers turned out easier to jailbreak than models hardened for Polish speakers, so they secured PLLuM themselves. The other reason to build, beyond quality, is competence. Some Llama versions and Kimi already write the EU out of the license. Markets close. Models get forbidden. "Sometimes it is better to have the competences to build even something a little bit worse, but have the ability to do it."

American labs are writing constitutions and model specs, and fighting over whether the result is too woke or not woke enough. Labenz asks whether Poland has an equivalent document. Kozlowski says not really, and not soon. They want models to be ethical, not to blame people, and to steer around hate speech — the same floor the US and Chinese systems claim. They do not currently ship a long list of behavioral points the model must obey. Poland, in his view, is a couple of years behind on that kind of value engineering. The live constraints are about *data*: what you may train on, not a fifteen-page theory of the Polish soul.

He is equally uninterested in deciding how Catholic the national model should be. A majority of Poles still identify as Catholic, but the urban-rural split is real and the share is not what it was. The lab's posture is liberal: they release chat models, instruct models, and base models, and let companies pick, because some constraints wreck business cases. Builders, he says, should not decide for everyone how to skew the system. People can preference-train a model into a narrower stance later. The harder problem is the one US companies can often litigate through and European ones cannot. The AI Act on general-purpose models, layered with national authorship rules, is less flexible than any constitution. Those documents can eliminate 80% of a training set. That, not a values memo, is what moves quality.

Chapter 2 of the AI Act, in force from August 2025, demands a model card: what data was used, how the system was secured, what resources went into training. Kozlowski thinks that is why certain Llama versions and Kimi forbid EU use — the vendors will not disclose the mix. If the license were not an issue, he would still split by task. Long-form generation, essays, emails, historical narrative: he would be very averse to a Chinese model, because censorship leaves fingerprints in long prose. Extractive and analytical work, pulling facts from documents, is a different risk. There he is open to using them. Web crawls remain the majority of even PLLuM's corpus; they add a science library and bilateral publisher deals, a minority. US crawlers, including Anthropic's, still hit Polish sites that disallow them in robots.txt, and fighting that in an American court is not a realistic Polish hobby.

The most operationally useful claim in the conversation is also the least discussed in Silicon Valley. On PLCC, Kozlowski's team watched some Anthropic Claude releases — he mentions Claude and Haiku — lose Polish language and cultural competence relative to the previous version. They saw the same pattern in parts of the GPT line: not going up, and in some releases going down. The mechanism is a trade-off, not a mystery. Labs read the paying market, pour capacity into software-engineering assistants, and niche-language generation is what gets crowded out.

Labenz had assumed frontier labs still wanted every token they could get, and might even pay for a few hundred billion Polish ones. Kozlowski's evidence suggests otherwise. If the next training run is a choice between more coding traces and another language that is not the revenue center, the coding traces win. For a Polish company that has wired Claude or GPT into its stack, the failure mode is ugly: a new release ships, the competency you bought declines, and you either switch vendors or roll back — if rollback is still offered. Deep cloud integration, in that light, is a bet that a lab with no Polish market will keep caring about Polish. Kozlowski's commercial conclusion is the sovereign-AI pitch in one sentence: putting critical work on a cloud generalist hands the capability curve to a team that does not report to your market.

Labenz has seen the same pattern in his own work: when inputs and outputs are boxed in and other guardrails sit around the model, a cheaper network with the right training matches a giant. Kozlowski puts numbers on it. Businesses and public agencies rarely need a ChatGPT that can do a thousand tasks. They need ten or twenty. If you write at least 1,000 instructions per use case — 3,000 is better — and SFT a smaller model, quality lands at or above what you get from few-shot calls to a huge cloud LLM. The instructions should be organic, or semi-automatic with a human in the loop. That is a dataset a determined team can actually produce. It is not a few hundred billion tokens.

On-prem is often not a preference. Some sectors are not allowed to use the cloud, or the risk is too high. Once you have bought sixteen GPUs and started paying for power, everyone discovers downscaling. Agentic setups make the pressure worse: a complex workflow may call a dozen models, and hanging a frontier checkpoint on each one will not survive the energy bill. Small localized models are, in his view, the shape of that future. PLLuM itself is a public consortium — first six institutes and universities, now eight — funded by the Ministry of Digital Affairs, aimed at legal, transparent, on-prem use in the public sector rather than at winning consumer chat share. Chatbots for municipal halls matter more, he says, than shaving another point off a general benchmark when the hall has no bot at all.

A more aggressive move is domain adaptation: continued pretraining on a company's closed corpus so the model learns the house dialect. They did this for PKO BP, the largest bank in Central and Eastern Europe, and some tasks moved a lot. The bar is high. After dedup and filtering you need around 10 billion tokens; that implies 30 to 40 billion before curation. Kozlowski estimates fewer than a hundred European firms have closed internal data on that scale. Labenz objects that a 40-person company, counting Slack, Google Docs, Jira, and contract revision history, feels like a lot of text. Count it, dedupe it, filter it, Kozlowski says, and 10 billion is hard. Sales and call-center mail can be usable. A great deal of other email is too sensitive on intellectual-property or cybersecurity grounds to touch.

The sentence Kozlowski most wants companies to remember sits late in the conversation. Firms jump to model selection and deployment, skip the inventory, and the project falls over. The order he wants is the reverse. Identify what data you have, what is high quality and what is junk, what internal and external rules allow you to use, and only then talk about training or serving. Even large organizations, he says, often do not know what is in their stores. Cleaning those stores is a problem in itself, not a preamble you can wave past.

That is also why he is cool on federated learning and cryptographic data unions, including the Near Protocol and Intelligent Internet schemes Labenz raises. The direction is right: anonymize, encrypt, share repositories, train without handing over raw files. In the economy he actually sees, most companies cannot measure their own in-house data, so they cannot join a mixture with anyone else. Distributed, secured repositories used by big consortia may arrive. They are not, he says, next year's project.

Kozlowski reads the last few years as a plateau. Reasoning is up, but GPT-5 over GPT-4 was not a rules-of-the-game jump in the way ChatGPT, GPT-4, and multimodality were in 2022–2024. Gains look horizontal. The explanation he favors is data: the largest vendors have already collected almost all the organic text the public internet will give them, and some are scanning books that never made it online. The organic well is close to full. Labenz is betting that synthetic data and simulated worlds will break the stall, and admits the bet is not settled. Either way, Kozlowski thinks the verification era will teach buyers they do not need huge generalists for most on-prem work.

If the jumps continue, and a $10 billion run really does beat a $1 billion run, his compass is still the customer's metric, not the public leaderboard. Most business cases he sees do not need a reasoning trace. They need an email that produces a cross-sell. Start from the task, write a benchmark for that task, then pick the model. Labs and business, he says, are two trains on parallel tracks; the crossing is still a long way ahead. On the international board he sees the same tension in another form. Fast products require centralization — Silicon Valley as 90% of the startups. Broad economic and social returns want distribution across a continent. You cannot run both strategies at once. China now pays researchers in the same neighborhood as the United States. Europe's public sector struggles to define the objective function at all: ten million customers, or $10 million a week in subscriptions, is not how a ministry is allowed to think, and overpaying niche talent like NFL quarterbacks is politically immature. Mistral, once the European village's hope, is in his telling 30 to 40% Microsoft-owned and less open than it was.

He is not despondent. Poland is currently pouring effort into AI factories — GPUs and data centers — and into a handful of PLLuM-like public projects, of which he thinks there are only two or three in the EU. American hosts at a Las Vegas leadership summit were surprised that a ministry would fund this at all. The value, for him, is not only the checkpoint. Even if the thing they ship is a bit worse than the Chinese or American one, owning the skill to make it means that at some future fork they can take another road. That option, he keeps repeating, can mean more than it looks like on a scoreboard.

Kozlowski closes by sending listeners to the November arXiv paper, *The PLLuM Family*, almost a hundred pages of data mix, training stages, and the rest of the recipe. For a country of 38 million, the document is a stance. The models are not huge. The team knows how they were built, and is willing to tell you how to build them. Weights without a cookbook, in this project, would have been a half-open gesture. The paper is the other half.

The operational list for an IT lead or a ministry is short enough to start this week. Inventory data before you shop for GPUs. Count whether domain adaptation can clear 10 billion clean tokens; if not, skip continued pretraining and write the 1,000 instructions. Regression-test every cloud bump on your own cultural and long-form suite. Define the business metric — the letter that sells, the extraction that is right — and ignore the general leaderboard until that test exists. And keep an exit from any lab whose next release might quietly forget your language.