The Transformer Co-Author Who Wants to Give AI a Wallet

Sarah Guo opens No Priors with a number: more than 25 million people already use applications on NEAR, which the show describes as a blockchain operating system. She and Elad Gil are talking with Illia Polosukhin, co-founder of NEAR and one of the authors of the 2017 paper Attention Is All You Need. Two other authors of that paper, Noam Shazeer and Jakob Uszkoreit, have already been on the show. At the end of the episode Guo jokes that she and Gil are collecting the Transformer authors Pokémon-style.

NEAR did not start as a chain. Polosukhin says he and his co-founder Alex Skidanov launched the company around 2018 to teach machines to code, betting they would ride an exponential AI wave they expected in 2017 and 2018. That wave did arrive, he notes, just not until 2023. In the meantime they assembled a community of mostly student crowdsource workers: give them code, get descriptions; give them descriptions, get code and tests.

The unglamorous blocker was payroll. Many of those workers were in China, Eastern Europe, and other places with capital controls and no bank accounts, so the company could not pay them. They looked at blockchain to solve that internal problem. When the AI explosion failed to show up on schedule, they decided to build a chain they could actually use, and wait. "As you go into the blockchain rabbit hole," Polosukhin says, "you realize there's a lot more that meets the eye." The thesis he wants listeners to take is simple: give an agent a blockchain account and it is no longer just a chat program. It can pay people and other AIs to work.

Guo asks how Polosukhin got onto the Transformer paper. He was on a Google natural-language-understanding team focused on question answering. The state of the art was LSTMs and recurrent nets, which he says you could not launch in production: they were too slow, and they fell over as documents got longer. Jakob Uszkoreit was already using attention for query similarity and floated using it between encoder and decoder. Polosukhin jumped in, and with Ashish Vaswani they tried a blunt question: could attention alone teach word order and do translation?

It could, and the resulting paper, data, and developer community became starting capital for NEAR. Calling NEAR a blockchain operating system, Polosukhin says, is about going up a layer. Users should discover and use Web3 experiences without thinking about the low-level implementations underneath, the way a phone OS hides networking and payments. Developers build apps; the platform is supposed to deliver them to consumers.

Gil remembers meeting Polosukhin just as the original AI mission was giving way to the chain, and remembers the team as one that could build almost anything. The original blockchain bet, Gil says, was tied to data labeling and payments. Now Polosukhin has been thinking again about where the two worlds overlap. That is the rest of the conversation: markets everyone already talks about, and a second layer he finds more interesting, in which language-model agents get accounts and start hiring.

Polosukhin splits the AI–Web3 overlap into layers. The obvious one is marketplaces for compute, models, and data: Web3 is good at creating markets, traceability, and a place where participants can share the upside. The more interesting layer is agents. Equip them with a blockchain account and they become economic agents that can pay other people and other AIs. People who treat language models as just another incremental advance, he says, miss the point: "this is the first time that a machine is able to communicate with people in the same way." There is no longer a human in the middle interpreting the data for everyone else. The machine can assign work and hand over context directly.

The use case he keeps returning to is an organization run entirely by AI, with the CEO role taken by an agent. A community, board, or other oversight body gives it a mission and KPIs, and can even feed it training data when it is off course. That layer, he argues, could eat a lot of middle management whose job is transforming information, handing people a slice of work, and folding their output back together. Gil asks why you would want blockchain and AI for something like commercial cancer research. Polosukhin's answer is coordination: cancer is many sub-cancers, each needing experiments, information flow, recruiting, candidates, pipelines, and funding. Grant allocation, nonprofit fundraising, experiment design, and sharing are, in his word, broken.

An AI coordinator could read all of that, pick the lab best at a given experiment, fund it, watch delivery, and fire it without personal affiliations. He expects simpler versions this year, with straightforward missions and KPIs, where onboarding and information flow already run through a language-model agent. Gil offers a nearer step: performance feedback from an AI rather than a human manager. Some surveys, he says, suggest employees may prefer it because it feels more objective and less personal.

Polosukhin thinks DAOs will show it first. Running one, he says, is a dull job: onboard new members, explain the same things, answer the same questions. A Discord bot with the DAO's full context can already do that and hand out starter tasks. Payments are native, and the social constraints of a company are weaker. "A lot of people will revolt if tomorrow you say, by the way, your new boss is this AI model." DAO members, in his telling, care less.

Asked how blockchain and AI meet on alignment, Polosukhin flips the frame. "We need human alignment instead of AI alignment." The problems people worry about, he says, exist because humans do things, and they existed before models. He reaches for Byzantine fault tolerance, a foundation of blockchain whose historical root is armies trying to stop people from propagating false orders. Misinformation is old. Society built some tolerance for it in the television and mass-media era; it does not yet have a system for the same problem at the scale AI enables. "That's what AI brings," he says: scale.

Reputation, identity, and the social operating system of communities matter more in that world, especially when a malicious actor can mass-produce personalized falsehoods or invent a political candidate tailored to each voter. Web3, in his view, is a set of primitives. Some phone cameras already have a secure enclave that can sign a photo; as the image is filtered, you can keep a proof of time, place, and processing. The same idea applies to this podcast: if the three of them signed the final file, a listener could check that it came from them and was not generated. Identity on-chain then becomes a coalescence of content and interactions, linked to reputation in different communities, so a viewer has some context for who is talking.

He wants a content version of SSL's green lock. When you open YouTube or the New York Times, you would see that the piece is signed by a party inside a trust graph you follow. Cryptography and blockchain are tools; the product still has to win. Government identity has to catch up too. The fact that a U.S. Social Security number is supposed to be secret and is something you hand to everyone is, he says, ridiculous. Gil notes that crypto already had programmatic actors around money-as-code, an execution layer without intelligence. Add models and you still lack a usable identity if an agent is going to represent you to another agent.

Polosukhin says identity has already arrived in a limited form: millions of people use chains for finance, and the wallet became an identity. Private keys are too hard a concept for most people. On NEAR you get a named account; his is `root.near`, which can hold many keys with different permissions. He can grant an agent a key to act on his behalf, revoke it, or give one to a specific app. What is missing is critical mass. If enough applications do not use the identity, it never becomes the default, the same uphill path SSL took until websites switched and browsers supported it. Then the transition flips: you should get it because it is easier, and because it also gives you more financial freedom.

Guo asks where reputation and content provenance are most likely to fail as models get more capable. Polosukhin points at the next year in the United States. Everyone, he thinks, will take whatever tools they have and use them, even just for kicks, and some players will be malicious. He expects fake narrative candidates, and he would not be surprised by a campaign page that logs you in and generates a custom agenda from your interests. New media outlets can be spun up to flood the zone. On the user side there is little framing: does this source have history, the right provenance, any validation?

Law enforcement is already behind. People are using the tools maliciously now, and police have no good way to tell if an image was generated. The same gap exists for audio. Someone can call you and play generated speech cloned from thirty seconds of recording. Guo calls it a huge consumer-fraud problem; Polosukhin says it is already a criminal one, with a very low barrier to entry. She names the path: upload thirty seconds to a service such as ElevenLabs, train a voice, and fool a financial advisor or a bank into moving money. Steal the phone and you can impersonate the person outright.

The hardware pieces, he argues, are mostly already on the phone: Face ID, fingerprints, secure enclaves that sign things and, as far as he knows, have not been hacked. What is missing is a product stack that pushes that authentication to users and into the apps they actually use. Without authenticated paths on calls and messages, he says, the situation goes sideways quickly.

Gil turns to a popular crossover story: crypto miners bought a lot of GPUs, Ethereum moved from proof of work to proof of stake, and those cards look, on a slide, like spare AI capacity. Polosukhin is unimpressed. The GPUs miners used are not the ones AI labs want. The field has zeroed in on A100s and H100s; mining cards are older and not as focused on floating-point arithmetic. CoreWeave is the example people cite, and its real advantage, he says, was not recycling miner silicon. It was knowing how to build data centers and how to get large allocations from NVIDIA. Older cards might help with smaller models or some inference. They are not a hidden training cluster.

Decentralized training across random GPUs on a chain has the same problem in a different form. Labs training large models sit on on the order of 800-gigabit links between GPUs. A typical home link is more like 100 megabits, when you have one at all. Until you can replay and work around that gap, he sees decentralized training as still not realistic, even if some research groups are trying. Inference is the interesting market. You train once: Llama is trained once, then everyone runs it everywhere. The economy of scale is on the serving side.

Two inference pains map onto crypto tools. Privacy: today you send input and output to a service that may or may not record them. Elasticity: if you have ten times more users tomorrow, you want capacity that can follow. Multi-party computation and zero-knowledge proofs, he says, can help with secure decentralized inference. That is more realistic than training, and more needed.

Data labeling is the job NEAR wanted to do first. Centralized firms such as Scale AI and Surge have grown large. Polosukhin still thinks a Web3 marketplace is more efficient in the long run: a global market anyone can join, with rules that pay if you follow them, without standing up local offices. He has used Mechanical Turk, where a client can simply decline to pay and workers have little recourse, while the platform also lacks real quality and knowledge assessment. Embed quality scoring and escrow in one open marketplace and both sides get something: workers get protection and anyone-anywhere payouts; task givers get a much larger workforce they can price themselves.

Quality is why serious labs still insource annotators for pre-training and RLHF. Polosukhin splits the difficulty into domain knowledge and validation. A centralized vendor that wants specialists often has to set up a subsidiary, hire, and train, which is a lot of overhead if you need developers to fine-tune a coding model. On a Web3 marketplace a student can join without a local contract. Students, he notes, often want the practice and the money. Task givers can specify cross-validation, including honeypots: planted incorrect answers that workers must mark wrong, or they lose a buy-in. That is explicit game theory. A contractor who slacks off usually has more leeway; a worker with money on the line self-evaluates harder.

NEAR both ships its own roadmap and spins work out. One project they are spinning out is that Web3 AI data marketplace, which Polosukhin says now has the pieces and is ready to go to market. The other near-term excitement is partnering with teams that already have users and want to open more functionality. Sweatcoin is his example: a Web2 step tracker with 120 million installs, used daily, maybe for a discount on the next pair of shoes. As it moves into Web3, users can join economic activity, learn what is happening in the ecosystem, take on tasks and gigs, and plug a once-closed points economy into a composable network.

The longer bet he calls the future of SaaS. A lot of SaaS, he argues, is one database plus a UI for one problem. The underlying store is not that different across CRM, hiring, marketing, and some project tools; stitching those databases together is endless work that always breaks. If the database is something the user owns, via Web3, you can build front ends on top from shared components or by describing interfaces and business processes in natural language. His sketch: when marketing drops a new creative, spin up a Twitter campaign and a dashboard that tracks conversions. The result is a generated, dynamic UI for that use case, with templates fine-tuned to the problem, and he says that is possible now.

Gil asks whether that world is traditional UIs, agents, or a hybrid. Hybrid, Polosukhin says. You still get a page of analytics a person can read, but you are also delegating a pipeline the way you would to a coworker. Teams might share one display of a business process, then fork it: one person cares about conversion, the other about retention. That dynamism does not exist when everyone stares at the same Jira board and half the tickets are irrelevant. Filters are not enough if you wanted different information shown a different way.

Guo reminds him he is an author of the paper that changed the field, and asks whether 2023 is still bigger Transformers all the way down. Polosukhin wants models that can think, or process more, before they output, and he thinks that still fits inside the Transformer. He has not seen another architecture that matches his intuition. The simplicity of this one, plus the amount of optimization pouring into it, will be hard to match. Expressivity is not the bottleneck: you can already express any function. The work is cleaner data, self-critique, and not forcing the next token when the model does not have an answer yet. That is engineering and more kinds of training tasks, not a new skeleton.

The simplest training trick he offers is to let the model emit empty tokens for a while and only then produce the next word, so it can be trained to think first. At inference you can vary the budget: more time to think, or none. You can also emit an answer, feed it back, and ask whether it is right. Jakob's point, as Polosukhin cites it, is that the model is already doing effective search in a knowledge space; pushing further there is more useful than bolting on searches at inference, because by then you have already lost the semantics.

Gil describes a lock-in: Transformers run well on the GPUs everyone is using, and so much software is being built around them that rival models will not get the same optimization. Polosukhin grants that another architecture might still fit the same silicon. He mentions, without naming it, a company whose chips are small compute tiles in a grid, with data flowing from one side to the other. The design is interesting, he says, and you can run Transformers on it, but the cost-to-output ratio is not compelling next to GPUs and the new accelerators aimed at this architecture. Without a lot of new models that fit that silicon, he would not expect a line out the door, and it is hard to match NVIDIA's economies of scale.

Gil compares the loop to 1990s Wintel, Windows and Intel optimizing for each other until mobile and ARM broke it. Polosukhin agrees: what changed that monopoly was a new form factor, and a full-stack bet on software and hardware is a lot of investment. Gil adds that the GPU supply crunch creates more incentive for heterogeneous hardware than there has been in years, though CUDA-level software may still win. Polosukhin's counter is that every large company, plus people spinning out of those labs, is building its own accelerator, and those chips are still optimized for Transformers or similar structures, hitting the market this year and next. Architecture, chips, and software reinforce one another until something hits a hard ceiling, or a new form factor shows up.