When AI Becomes the Oil Well: Luke Drago on the Intelligence Curse

On the Future of Life Institute podcast, host Gus Docker asks Luke Drago to define the idea behind his essay series. Drago, co-founder of Workshop Labs and co-author of *The Intelligence Curse*, boils it down: if non-human factors of production become your dominant source of output, your incentives are not to invest in your people. The name is borrowed from the resource curse. States that live off oil often put the next dollar into the well rather than into schools and industry, because the well pays more.

The result, he says, is a worse quality of life for people who now have much less power. At the core of that power is a mechanism that is easy to skip past: your ability to produce value is a core part of your bargaining chip in society. Docker restates the worry as governments and companies pouring money into still-more-advanced AI instead of into workers and citizens. Drago agrees. That is the curse in one move.

Economists often reply that every past technology raised wages and living standards after a messy transition. Drago treats that as a category error. The last thousand years of tools, he says, were extremely adaptive for humans. They helped people do new things and did not encroach on the advantage that kept us at the top of the food chain: the ability to think and then act in the world.

The Industrial Revolution did automate large parts of physical labor, and people feared they would stop mattering. No machine from that era, he notes, fully automated human thinking. Look at the stated goals of many labs today and you find the opposite bet: they stake their existence on systems that can do everything a human can do, better, faster, and cheaper.

He is careful not to cast this as an evil plot. If you can save fifty percent on the wage bill while getting better, faster, more reliable results, most people will take that option. As technology designed to replace rather than augment keeps advancing, he worries, we move toward a world where people just do not matter — and then quality of life depends on government after you can no longer produce it yourself. That, he says, is a precarious place to stand.

Docker offers the obvious comfort: pensioners produce little and still draw on society’s resources, so why not expand that system into universal basic income and a rights-protecting state? Drago’s reply is that economic value is an important part of political value. Democracies often start when diffuse actors with capital need a way to settle disputes short of violence. British democracy and Magna Carta, he says, grew because lords had material power that was not equivalent to a king’s but still had to be bargained with.

Economic liberalization, in this telling, is often a precondition for the democracies people actually care about. Non-democracies can be livable, but governments that respect rights and enable prosperity track closely with democratic ones. Flatten the underlying economic structure that created those bargaining chips, and you shrink the chips themselves.

Pensioners are not people who appear and never work. They spent forty years paying into a system as active members of society, then cashed an exemption in the last decades of life. The United States treats retirees far better after the New Deal than it did before, and that settlement involved mass unrest and workers using their bargaining power. Drago’s fear is a world in which we are all pensioners forever, with no way to bargain, at the mercy of the next election.

Asked which metrics would confirm or kill the hypothesis, he lists income inequality, a sudden compounding of capital once talent drops out of the loop, and falling economic mobility — the American story that you can start from nothing and still have a path. He also watches unemployment in the earliest age brackets just entering the workforce. Those, he says, are the numbers he has told others to track.

The essay series opens with a blunt prediction: if the trend continues, you are going to lose your job. The story Drago tells is pyramid replacement inside a multinational white-collar firm. Every year those companies hire a large analyst class from top universities, because the org chart is wide at the bottom and narrow at the top. People at the peak retire or leave; the funnel has to keep feeding leadership.

AI, he claims, first makes it easy to replace the people at the bottom. Entry-level work is standardized, checkable, and tolerant of mistakes — the kind of task you train and deploy against. A paper that had just come out, which he flags he has only skimmed, finds that in some fields AI looks augmenting and in others replacing. In targeted areas including software engineering, job postings, offers, and overall employment had shrunk in the 22-to-25-year-old bracket. That is exactly what you would expect if entry-level work is easiest to automate first.

The claim is that the model then climbs. As systems get more agentic, handle longer-horizon planning, and absorb more of a firm’s internal knowledge, replacement moves up rather than middle-out or top-down. One day you wake up and your colleagues are AI; the next knock is for you. He thinks this can happen at every level of a white-collar firm, with the usual caveats that industries and internal functions will not move in lockstep.

Blue-collar work, he guesses, looks more zero-to-one: there are not yet robots that can do the jobs, and then there are. Those companies are flatter, with many people doing similar work and a small pyramid on top, which would make a robotics breakthrough concentrated and ugly. Docker adds a twist: managers of physical workers may go before the workers, because invoicing and scheduling are already automatable while the robots that move boxes are not. On whether falling programming postings might just be rates or a market cycle, Drago says the test is whether the hit lands on tasks we already know are automatable. His skim of the paper, he says, roughly matches that map.

Docker asks which jobs sit behind legal walls. An AI judge employed by the government, he suggests, is among the last roles anyone will automate. Drago points to work by Derek Chang — now at the Windfall Trust, previously at Convergence Analysis or Convergence Research — on which jobs look more resilient. Physical labor is still harder for now, reversing a fifty-year story that machines would hit bodies first and minds second.

Strong legal protections do make a job harder to automate. They do not stop the people in those jobs from automating themselves. That is both an opportunity and a path to gradual disempowerment: you keep a human on the masthead while a generic model makes the decisions. A world in which every judge uses the same model and the same prompt, Drago says, would be a bad one. You would at least want diversity that tracks the actual beliefs of the people on the bench.

Partner-level lawyers will be hard to replace; first-year associates do mostly grunt work. If a firm hires half as many of them, lawyers have not been automated on paper, but the junior pipeline has collapsed. The fork he and co-author Rudolph care about is what happens next: do large firms keep accumulating capital without hiring, or does an abundance of smaller specialist shops appear? They are much more excited about the second world. They do not think it is the default.

Docker underlines the formal-versus-actual split: society could keep the office of judge and never wrestle with whether it wanted to outsource judging. Drago’s sharper worry is the information inside the automation. If everyone is calling GPT-7 to do the judging, then whatever flaw lives in GPT-7 is now your judge.

Docker wonders whether human judgment and taste could remain a job: hundreds of models can generate writing or images, but picking what people actually find interesting is harder. Drago says it depends on the pace of capabilities and on what we aim for. He is much more excited about a world where diversity of taste stays a durable human advantage.

His favorite example is Nomads and Vagabonds, an artist on Twitter who made the art for *The Intelligence Curse* and for Workshop Labs. The workflow is a Stable Diffusion model fine-tuned on the artist’s own work, unusually good prompting, hundreds of generations, and a very small number released. Drago has tried the same stack and cannot get the result. He does not know what data to feed, how to write the prompts, or which few images to keep. Nobody, he says, could look at that output and say it belongs to anyone else.

That is not, in his view, the goal of the major companies. OpenAI’s definition of AGI is predicated on doing most economically valuable human work — a different game from tools that let you change the world while leaving some of the work in your hands. The target right now is total automation. Docker then turns to a barrier named in the essays: local and tacit knowledge, spread out, hard to formalize, and constantly shifting.

Workshop Labs’ thesis, in two sentences, is that the bottleneck to long-term AI progress runs through high-quality data on tacit knowledge and local information. Tacit knowledge is the feel you accrue by doing the work; it is hard to digitize not because it is mystical, but because only you have it and outsiders do not know where to look. Local information is what you see because you are an embodied person with real-time access to your sphere. Labs want that data — hence the rush into the browser, and the expensive custom RL environments built with experts. Drago’s pitch is that the valuable data is already in your pocket, on your laptop, and in your day. Workshop wants to put it to work privately, so the company cannot train a larger model to automate you or sell the data to your boss.

Docker sees a new labor fight: workers refusing to hand over tacit knowledge, leadership eager to train on it and cut costs. Drago says that is one tension, and then points to a fact people forget: fifty percent of Americans work at a small or medium-sized business. Those firms do not have hundreds of people to mine for surface data. Most people on the team actually matter; if they did not show up, something would break. Their process knowledge is the company.

The outcome he wants is AI shifting advantage away from extremely large companies, whose tasks are already automatable, and toward an explosion of small enterprises that understand what is going on locally. Intelligence put to work for the average person could close the efficiency gap that scale usually buys. That only holds if the things that make you competitive are not given away. He calls data the new Social Security number. In 2015, caring about privacy mostly bought you worse ads. In the next ten years, handing a lab your life for moderately better ChatGPT results — if they do not lock it down — is one button away from someone hoovering it up, selling it, and automating you out of the economy.

The individual incentive still cuts the other way. Docker’s example is a maths PhD on a low salary offered hundreds of dollars per step-by-step proof. Drago does not think “worse but private” wins that game. People will not choose a worse tool for privacy. The default tool has to be better at your work because of the data it has, then layer on extraordinary, verifiable protections. He cites Apple as a privacy-second company that still locked down messages and devices because it is infrastructure, and Anthony Aguirre’s writing on loyal assistants: you have to know the model organizing your life works for you.

Asked how to guarantee privacy without trusting Workshop, Drago points to encryption in transit, decryption inside a trusted execution environment using NVIDIA Secure Enclaves, remote attestation of the running code, and encrypted model weights. You should be able to see that nothing is being extracted. He then sketches the bad scenario if the curse lands: a 2030 computer-science graduate who cannot find an entry-level job, strained unemployment, Microsoft-style record earnings from halved headcount, and a federal budget in which about fifty percent of tax revenue still comes from income tax while the wage base shrinks. Safety nets fail, unrest rises, a thin UBI does not hold, and a few companies decide government is mostly in the way — a ripe setting, he says, for the kind of power grab described in work by Tom Davidson and others.

One necessary condition for breaking the curse, Drago says, is commoditizing the intelligence layer. If one, two, or three players monopolize intelligence, they can keep raising the rents. He recalls a tweet: wrap a commodity and you are a landlord; wrap a monopoly and you are a renter. Cheap, prolific intelligence that you specialize to your own work is a better world. Lab self-improvement takeoff is a different game — a few winners. Commoditization does not fix everything. He still thinks it is a precondition. And the buyers in a thin consumer economy, he adds, can just be other firms: land, compute, energy, intelligence trading in a closed loop, with far fewer cafes and far more data centers.

The resource curse is not guaranteed doom. Norway has a sovereign wealth fund, a real economy on top of oil, and, by many metrics, excellent education, services, and a stable democracy. The load-bearing fact in the case studies is that Norway had extremely resilient institutions *before* oil: a capable civil service and very low corruption. Drago’s question is whether we live in that world now. He does not think so. Almost every American who has looked at government, he says, thinks something is fundamentally broken, and has thought so for decades. Withstanding total automation would take more institutional resilience than withstanding oil money.

Saudi Arabia and the UAE are the other comparison: peak-oil anxiety and a push to diversify. High quality of life there, he notes, is for people who are economically important to the state; there is still an underclass. As Saudi has moved toward a more diverse economy under MBS, it has also somewhat liberalized gender relations. Those things happening together is not an accident. Arguments that we can centralize power, automate the economy, and still expect the state to care about you should be measured against Norway versus states where you are easier to ignore if you are not useful.

Historically, rulers share rents when a credible threat of revolution is more expensive than paying out. AI that can automate every job can also automate repression and surveillance. Make society more legible and dissent is easier to crush. State capacity, in his telling, goes through a weak period and then, once repression is automated, becomes suddenly very strong. In both phases, democratic process can stop working. Incentives are powerful, he tells Docker, but they are not predetermined or ironclad. George Washington stepping down rather than making himself king is the classic defiance. Brexit is a society choosing sovereignty over a strong economic argument. He quotes Charlie Munger: show me the incentives and I will show you the outcome. If you can move those incentives toward humanity, you should.

Asked where to push talent under differential technological development, Drago lists three buckets from the essays. First, defensive technologies that actually de-risk AI, because catastrophic risk is a credible argument for centralizing intelligence in a few hands. Kyle O’Brien’s work, he notes, found that stripping biological-materials information from pre-training data can leave models somewhat tamper-resistant even when you try to put that knowledge back in at fine-tuning. That is the kind of result he wants more of: if we build the systems, they need not live with one controller forever.

Second, technology that keeps humans in charge of the last mile — putting an individual’s data to work so the tools they control make them more competitive. Third, technology that strengthens democracies, in a vein he finds inspiring in Audrey Tang’s vision. He compares the pair of strategies people use on social media: regulate addictive features, and also ship tools that point outward, like screen-time apps. Workshop is trying to be the outward-pointing tool in a market of systems trying to take your job.

On fully open-sourcing AI, Drago says he is probably more pro-open-source than the average listener on the podcast, because of the monopolization fear. If open-weight models are not a core part of the future, closed providers can charge wild rents. The popular safety-community claim that open weights will always lag is, he thinks, wrong except in a hard takeoff that fooms to superintelligence — in which case someone wins and the game is over. In almost every other scenario the gap has closed. A few years ago people said China could not catch up; Chinese open-weight models now sit about six months behind the frontier, and he would wager Kimi-K2 is state of the art at English writing. Compute is not everything. If you take safety seriously, you have to make open-weight models safe, because they are going to exist and they are going to be powerful.

You cannot take an open-weight release back, Docker objects. Drago comes back to tamper-resistance as a holy grail: a model that breaks or stops working when you try to reintroduce dangerous knowledge. He is unconvinced by the plan of evaluating until a catastrophic warning shot makes everyone stop. *The Intelligence Curse* has a seven-paragraph footnote arguing that in many scenarios a warning shot *speeds* AI progress, because the spooked response is that we need better defenses faster. Money and talent, he says, are better spent on making even open weights tamper-resistant. On the one-lab-controlled takeoff versus many, he would rather many. Aligned superintelligence in one person’s hands makes that person a de facto dictator unless they choose not to be. Workshop is a public-benefit corporation whose fiduciary mission is not to automate people — lawyer-speak for enhancing economic opportunity — and it plans to launch with privacy guarantees you can check rather than a promise you have to trust.

Docker puts the loyalty problem in product terms: a personal agent that books hotels may also carry a corporate preference for a partner chain. Drago’s rule is that a model should either answer in your interest or tell you when it is not. If LLMs go down the advertising path, ads should be as obvious as they once were in search. A *Black Mirror* episode sticks with him: a woman with half her brain in the cloud stays alive, then blackouts into ad pitches she cannot recall, then faces rising rents because one vendor owns the chip. Democratize the tech, keep the agents loyal, and, with Sam Altman, treat something like AI privilege as real — a system that organizes your life should not be usable against you, and it does not currently have a Fifth Amendment.

The default future, Drago agrees, is ad-supported and hard to distinguish from genuine advice. That is why Workshop exists. He keeps returning to Apple: the company makes money on the device you want to work, and native ads on personal devices have mostly failed even when Amazon tried them on the Kindle. Users handing over job-critical data, in his early conversations, are already more skeptical unless they know the tool works for them. Being that provider is both an impact thesis and a market.

For young people, his line is hard: the default paths are closing, whether or not any one intervention — including his company — works. Consulting jobs still go away. Fortune 500 pipelines that parents paid tutoring for are now among the *least* safe options, because a firm with hundreds of thousands of people on payroll wants to cut it. If you are N-of-1, doing a job nobody else can replace by virtue of being there, you are much safer than if a thousand colleagues do the same thing.

Automation of rote work can start total pyramid replacement, or it can open a more local, individual economy. The second world does not happen if you do not change path now. He is especially pointed at prestige tracks: straight A’s, the right college, Jane Street or McKinsey at the door with a massive paycheck. Take it, he says, and you are on the last chopper out of Saigon — the last breed of consultants, the last breed of entry-level whatever. His urge is to take more risk while the window is still open: start the company, join the unheard-of shop, do the moonshot. Guard the tacit knowledge in your pocket, and prefer tools that can *prove* they cannot use your data over tools that *promise* they will not.