The Transformer Co-Author Who Says Scaling Hit a Wall

Jacob Efron opens Unsupervised Learning with Aidan Gomez on enterprise adoption, data labeling, and whether transformers should still be the last architecture standing. Gomez co-authored *Attention Is All You Need* at Google in 2017, the paper that became the base of almost every large model. He now runs Cohere out of Toronto, selling into the enterprise — including AI features inside Oracle's Fusion suite — and shipping an agent platform called North. The hour is dense: which enterprise motion wins, which use cases already pay, whether scaling has stalled, how much human data still matters, and what might replace the transformer.

The line he is most embarrassed by as an industry sits later in the conversation, and it is the right place to start. Labs spend on the order of $100 million to train a model, then freeze the final checkpoint and hand the same weights to everyone. The system has read the internet and can reason through a proof. It does not remember last month's chat. You correct it, close the window, and the next session starts from zero. Gomez calls that a missing property of intelligence: humans start dumb at a new job and become experts over four or five years of practice. Models should learn from experience and from the people they work with. Until they do, the $100 million genius has amnesia.

Efron sketches three deployment models he sees in generative AI. One is straight consulting: a large Accenture team comes in and builds. One is Palantir-like, with forward-deployed engineers and a product being built alongside the customer. One is out-of-the-box software: here it is, you make it work. Gomez's long-term bet is the middle. The technology is new and complicated. He wishes enterprises could adopt it without help. They often cannot. Some degree of support, he says, is going to be necessary if this is going to land across companies and applications.

The reason is what an agent actually is. It can drive automation only if it has access to the same things the humans have: email, chat, calls, CRM, ERP, HR software. That much context creates two problems almost no traditional enterprise product has. The first is privacy. Very little software needs that degree of access, so privacy is a bigger issue here than in the rest of the stack, and Gomez says it is something Cohere has worked to do uniquely well. The second is integration. Every company runs a different mosaic of tools. There is no standard setup, and each one needs custom work to pull the context into the model. North is being built to make that tapestry easier to wire up.

Efron asks whether future agents will eat the complexity: type in the tools you want, and the system installs itself. Gomez wants that to be true and does not think it goes all the way. There will be a middle ground that automates parts of setup, not the whole thing. The stakes of a mistake on salary data, customer data, or patient data are high enough that you want guardrails and humans in the loop. Self-serve is a direction, not an endpoint.

Silicon Valley argues about experimental budget versus real budget. Gomez sees a lot of paid use. Some of it is vertical. In healthcare, Cohere wants note-taking and form-filling to get easier for doctors: passive mics on the doctor-patient conversation, forms pre-populated, so clinicians are not spending half their time typing. More of what he sees, though, falls into general categories that cut across industries.

The first is customer support. The technology is ready, the need is obvious, and it is moving quickly in telco, healthcare, and financial services. The second is research: an agent that can do a month of work in an hour or two. His example is a wealth manager with 20 to 50 clients who gets a call asking to hedge a geopolitical event that might be days away. The model can read hundreds of thousands of times as much as a person and come back with a cited report the human can audit. Gomez's line is that this can make knowledge work ten times more effective.

Looking out twelve months, he thinks deep-research applications are ready for prime time. A report that used to take a month and a half from an expensive person now comes back in 30 minutes to an hour, and enterprise buyers find it surreal. As infrastructure like North gets installed — access to tools and context, plus a UX for asking the model to accomplish a task — more mundane back-office work comes online: finance, legal, and especially sales. What a seller does before a meeting is research: the account's strategy, the decision-makers, prior conversations, where things stuck. An intelligence briefing, Gomez says, makes you dramatically more effective.

Efron asks whether enterprises will keep buying best-of-breed point tools — Sierra for support, Clay for sales — or consolidate onto one platform. Gomez expects a scattershot phase, then a squeeze. Sellers do not live only in the CRM; finance does not live only in the ERP. Each point app has to maintain its own data connections, and the maintenance burden pushes companies toward one platform plugged into everything. That long game is what Cohere is building with North.

Efron asks whether a frozen snapshot of today's models already hides a trillion dollars of value, or whether the curve of improvement still has to run. Gomez is almost impatient about reasoning. The input space is language, which is everything from a chatbot answering 1+1 to proving a theorem. You should not expect the model to spend the same compute on those two questions. Reasoning is the capability that lets the technology spend different amounts of energy on different complexities of problem, and he thinks it is obvious the stack has to have it.

Cohere does not care about math olympiads. Enterprises are not begging for that. They are begging to automate back-office processes that jump across software and out onto the web. Pre-reasoning, there were tasks the model almost always failed. With reasoning, it almost never fails. The unlock is reflection: understand why the first attempt failed, then find another path. Gomez likes reading the traces. The model has little epiphanies — it is not over there, maybe because of this, I should look over here — and that, he says, has been a complete step change.

Scaling is the belief he has actually changed. For a few years he stayed loyal, even with a quieter doubt that every needed capability would fall out of a bigger run, because the evidence kept showing up: make it a bit bigger, now it can do math. Now he thinks they have been beaten over the head with the opposite. 'Scale is all you need' is breaking, capital and compute are in diminishing returns, and the old strategy was boring: build a 2x larger computer, press the same button, watch the loss drop. Test-time compute still makes inference three to ten times more expensive, but cheaper flops and more chip options are, for a lab that consumes a lot of them, a good trend.

The missing piece that would change the product is learning from experience. Right now feedback dies when the chat dies. If the model were an intern, you would teach it, it would stop making the same mistake, and you would have turned it from a first-week hire into you 2.0. Efron calls the current loop Groundhog Day: a new batch of interns every week. Gomez does not think the mechanism exists yet. His guess is a queryable history of interactions, always in context at generation time. People are trying stuffing the window, databases, RAG. Once it works, he expects a different kind of investment in the system, because it is growing with you.

Efron asks whether the world is heading toward one model to rule them all or a zoo of specialized ones. Gomez used to lean harder on specialized models. Mixture-of-experts architectures let a network grow its own sub-experts, which relieved some of that pressure. Custom models still matter where the web is silent: manufacturing data, customer transactions, detailed personal health records. Cohere partners with the organizations that hold that data and trains a model only they can use. General models plus synthetic data close a lot of the rest of the gap. He does not expect tens or hundreds of models inside one company. A handful, maybe. Not a fine-tune per team.

On data, the crowd-sourcing era is over for the hard domains. Finding 100,000 average people to teach a model to chitchat was viable. Finding 100,000 doctors to teach it medicine is not. The new move is a much smaller pool — on the order of 100 doctors — who provide lessons and demonstrations, then use that trusted seed to generate a thousandfold of synthetic lookalike data. Code and math are easier because you can check results and filter garbage. Other domains are harder and still viable. At this stage, Gomez says, an overwhelming majority of the data Cohere generates for a new model is synthetic.

Evaluation is the side where you cannot take the human out. If you are building models for people, people are the ones best seated to judge usefulness. The way to remove the human is to have an expert better than the current model watching it, which assumes you can already build that expert — the thing you are trying to create. So eval still has a hard dependency on humans. Test-time compute, for Cohere, is not about olympiads. It is about teaching models to reason through the problems businesses already solve with the tools humans already use.

North is Cohere's push into the application layer, and the motive was repetition. Customers kept building the same thing: a year spent on a UI over models, tool and data connections, privacy and ACL inheritance. The builders were usually the internal AI team, not a product team, so the experience was not something employees wanted to use. North is meant to be consumer-grade — intuitive, low latency — with the enterprise knobs: rebrand the UI, choose data sources and tools, even plug in a Llama fine-tune. Gomez's claim is that this sets a company forward 12 to 18 months on its product roadmap and puts the tech in employees' hands immediately.

The strategic attributes he leans on are not being locked into one hyperscaler's ecosystem, deploying anywhere, and releasing weights non-commercially, which makes VPC deployments easier because they are not afraid of a customer seeing the weights. The deeper argument is vertical integration. The latest Command model is already optimized for the North use cases. It knows how to use an ERP and a CRM. Application-only shops grab Llama weights and cannot change the technology for the customer. That, Gomez says, is why a lot of existing apps miss the product promise. If you look at the teams shipping the best products, they have a deep familiarity with how language models are built. Even when they are not training, they try to approximate impact at the model layer. Cohere tries to play that role for partners such as Oracle.

Google Brain, where he came from, was a home for brilliant people with complete research freedom, tons of compute, and great colleagues. That setup produces interesting research. It is not the setup that delivers a product. Cohere keeps the people, the compute, and the world-changing ambition, and drops the freedom to work on whatever you want. The objective is specific: make the technology good at using enterprise software. There is not time for unfocused exploration. The culture he wants to preserve from Brain is the warmth — a place that cared about something.

On the five-year map of foundation-model labs, Gomez sees strategies diverging: OpenAI on the consumer front, Gemini and Meta chasing that crown, Anthropic best at code, Cohere on enterprise back office. Among those companies he expects a couple of handfuls, then a new generation of models for biology, chemistry, and materials. He sits on the advisory board of a cancer data and compute sharing alliance that includes Fred Hutch; that data is not on the web, and the bottleneck is not token scarcity. The records exist in sufficient quantity, siloed in a hundred places that refuse to share — a human problem, not a generation problem. He welcomes new labs such as SSI and Thinking Machines and wants more companies to try.

As a co-author, he claims to have been the first person to ask in public why the transformer is still being used. Cohere's New York office has a meeting room named SSM, because he thought state-space models would kill the architecture. Then people stole the good bits of SSMs, put them into transformers, and the need to swap went away. Discrete diffusion is the new cool UX: a wall of noisy tokens, the answer emerging from the ether. He does not see why that would be a better language model. Asked in 2018, a year after the paper, how likely it was that the field would still be on transformers seven years later, he would have put the probability close to zero. He will not make a new estimate. He would like a successor, sincerely.

Keeping the company in Toronto was a 51-49 call that has paid dividends: Hinton is there, Ilya Sutskever came out of the ecosystem, and the talent is real. Internationally Cohere would rather partner than watch every country stand up its own lab. Gomez holds British, Spanish, and Canadian passports and talks about deep work with Fujitsu on Japanese and with LG on Korean. Cohere For AI's Aya project, which he calls the largest data-collection effort for any machine-learning project, had native speakers of more than a hundred languages contribute data and then open-sourced it so every model, not just Cohere's, got better. The tech is not useful, he says, if huge swaths of the world miss it because it does not speak their language or understand their culture.

He is not living differently because timelines moved in. His father is a cancer survivor, and he wants treatments cheaper and better; he also does not buy mass unemployment, because the world is supply-constrained and humanity has infinite demand if people can be retrained. He worries about bad actors at the state level and wants liberal democracies to get the capabilities first, but Terminator breakout is not where he wants policymakers focused. There is more than enough to get right in the near and midterm. His own test of a new Command model is meeting prep inside North, wired into every call and DM; third-party models he tries to break with character-level riddles, now that math is too easy to be interesting.