The Man Who Named AGI Wants Superintelligence Raised on Compassion

Lex Fridman sits down with Ben Goertzel, one of the more restless figures in artificial intelligence: founder of SingularityNET, designer of the open-source OpenCog framework, former research director at the Machine Intelligence Research Institute, and chief scientist at Hanson Robotics, the company behind the Sophia robot. The conversation runs more than four hours. What Goertzel wants from machines is not a better chess engine. He wants systems that can learn compassion, end involuntary death, and then go far beyond the human — and he has been arguing that case since long before AGI was a marketing term.

The obsession started with television. Watching original *Star Trek* with his father around 1968, he saw Spock and Kirk freeze a supposedly cosmic-scale robot with a logical paradox. The scene made him angry. A machine that had traveled the universe should not die on a schoolboy trick; if a human brain can step around a paradox, why couldn't it? He complained to his father, who did not settle the argument, but the complaint stuck. Years later the same father was running Fortran on punch cards at Rutgers, and middle-school Goertzel was doing linear regression on demographic data. His grandfather, a physical chemist who had worked among early quantum experimentalists, handed him mechanics texts at seven. The boy reached wave functions, understood that his grandfather did not fully understand them either, and was pointed to a professor who did.

Douglas Hofstadter's *Gödel, Escher, Bach* finished the conversion. Goertzel was about twelve when the book appeared. He read it in a sixteen-hour sitting and then read it again, this time for the formal systems. Until then, superintelligence and immortality had been science-fiction furniture — he had even toyed with building a near-light-speed craft, flying away, and returning when Earth had the technology. Hofstadter made the work look like programming rather than prophecy. Smart people at universities were already trying to build what Goertzel would later call AGI. You did not have to wait, and you did not have to leave.

Around 2001 or 2002, Goertzel began editing a collection of serious papers on machines that could think as broadly as people, or more broadly. The working title was *Real AI*. He recruited people he already knew from transhumanist and singularity circles: Peter Voss, Shane Legg (who had worked at Goertzel's late-1990s company Webmind and later co-founded DeepMind), Marcus Hutter, Pei Wang, and several Russian researchers. Narrow machine learning on genomics or chat sentiment was real work, he thought, but Ray Kurzweil's split between "narrow AI" and "strong AI" felt wrong. Strong AI already meant something else in cognitive science: the claim that a digital computer could be conscious.

On a private email list they tossed around names. Artificial general intelligence won, in part because psychology already had a *g* factor for general intelligence, so "artificial GI" sounded native to the field. Pei Wang wanted GAI, the Chinese word order, but they judged that it would not fly in American culture. After the 2005 book appeared, Mark Gubrud told Goertzel he had used the phrase in an essay around 1997. Goertzel says he is still waiting for a Russian to produce a 1953 paper. The coinage was obvious in hindsight, not a conceptual breakthrough.

He has never fully loved the term. *Artificial* makes intelligence sound like a tool, and not every AGI will be a tool; the long-term aim is agents, not appliances. *Natural* versus *artificial* is a false split if humans and machines are both part of nature. *General* is not quite right either: Hutter's AIXI is a beautiful picture of truly general intelligence inside computable environments, but it needs infinite compute, so infinite generality is not a physical object. Joscha Bach's *synthetic intelligence* is closer to what Goertzel prefers. None of that mattered sociologically. People with a technical background hear "AGI" and know what is meant. "It's like KFC," he says. The industry will probably keep the name until AGI systems take over and rename themselves.

Ask Americans about AI history and they will narrate summers and winters. Goertzel thinks the metaphor is provincial. Publication in the field since the 1950s shows fairly steady growth. What rose and fell was how much money the U.S. military — then the main patron of American AI — was willing to spend. Work continued in Germany, Britain, Japan, and Russia with less cash and more theory. The United States was unusually good at turning papers into running systems, and that implementation wave did track Pentagon enthusiasm. It did not define the global subject. In the 1980s, he notes, researchers in Germany already had self-driving cars on the Autobahn, years before the auto industry was ready to care.

When he finished a mathematics PhD in 1989, academic AI looked to him both boring and wrong. The fashion was rule-based expert systems and production systems. He had no quarrel with mathematical logic as a cognitive engine. He thought it was absurd to type in, by hand, the knowledge a thinking system would need. The machine had to learn from experience, and the mainstream was not interested. The people who *were* interested looked like isolated eccentrics. One of them was John Andreae, whose PURPOSE system treated AGI as fancy Markov-style reinforcement learning: bits in, reward maximized, intelligence supposed to appear. Computers were not ready. The ideas were already close to what labs do now.

That lineage is not folklore. Andreae supervised John Cleary in New Zealand. Cleary later worked with Goertzel at the University of Waikato in 1993, alongside Ian Witten, and they launched Weka, the first widely used open-source machine-learning toolkit — a predecessor, in spirit, of TensorFlow and PyTorch. Amusingly, much of the Waikato work was funded by a New Zealand government grant to predict the menstrual cycles of cows. "In the US, all the grant funding for AI was about how to kill people or spy on people. In New Zealand, it's all about cows or kiwifruit." Shane Legg passed through the same Waikato group and through Webmind, which is how AGI got into the head of a future DeepMind founder. Goertzel convened the first AGI workshop in 2006 in a Marriott in Bethesda, about sixty starry-eyed people who felt as if they were plotting against the narrow-AI establishment. Today the annual AGI conference is the main gathering of researchers who want human-level systems and do not believe large-scale nonlinear regression is the golden path.

OpenCog began as proprietary code inside a government-facing company around 2001, then was cleaned up and open-sourced in 2008, mostly in C++. Goertzel is careful to split the project in two. As a software platform, OpenCog could host many architectures. As an AGI design, a smaller group — including Goertzel and longtime collaborators — has been using that platform to test a specific theory of mind. The core store is AtomSpace: a weighted, labeled generalized hypergraph that lives in RAM, can spill to disk, and can be spread across machines. Links may join more than two nodes, and links may point at other links, so the structure is closer to a metagraph than to an ordinary graph. Nodes and links carry types and numbers: truth values, importance values, whatever the moment needs.

What captivates him is not the graph for its own sake. It is a common representation underneath abstract declarative knowledge, sensory knowledge, movement, procedures, and episodic memory. If a logic engine, a neural net, and an evolutionary program can only swap final answers, they remain a pipeline of black boxes. If they share intermediate state in the same form, they can pull one another out of combinatorial explosions. The practical trick was to stop writing logic as nested predicate-calculus quantifiers and to use Aristotelian term logic instead: concepts as nodes, relations as links, deduction as a path that can also carry a neural weight. A heavy attentional link between A and B can nudge the logic engine to ask what the relation is; a logical inheritance can nudge the neural side to spread attention. The Curry-Howard correspondence is supposed to do the same job for programs and proofs, mapping a robot-arm procedure into logic, improving it, and mapping it back before the expression bloats.

The first attempt, Webmind in the mid-1990s, was a cautionary tale: a giant in-RAM graph in Java 1.1, every node with its own little processor, a main loop walking the whole network. Logic, neural nets, and evolutionary learning were all present. The math that would let them cooperate was not. Decades of cleaner scheduling and representation still have not produced an AGI, though the stack has paid the bills in NLP, robot control, and finance. The rewrite now underway with Alexei Potapov's St. Petersburg team, provisionally called True AGI, keeps the paradigm and replaces the 2001–2008 infrastructure. Three bottlenecks drove it: faster checking of probabilistic dependent types; real-time interoperation with deep-learning compute graphs such as PyTorch's; and a hypergraph that can live across thousands of machines rather than dozens. Commercial AI, Goertzel says, is barely thinking about representation at all.

Asked whether GPT-3 understands, Goertzel does not hedge. "No. It understands nothing. It is a complete idiot." Size will not buy understanding any more than a faster car will get you to Mars. Transformers recognize a huge number of shallow patterns. They do not form abstract representations. That does not make them useless. As an entrepreneur he sees valuable products; as a musician and collaborator on Hanson robots he likes watching a model trained on Philip K. Dick emit stoned-philosopher lines. The mistake is to confuse that fluency with a world model.

His team's unsupervised grammar-induction work made the distinction concrete. You can train a matrix that maps a transformer's internal state onto the constituent structure of a sentence — Christopher Manning and others have done related probes — but the recovered representation is "horribly ugly." It does not look like the grammar rules you know are right. The method that works better is inverted. A symbolic grammar learner proposes a rule, generates sentences that obey it and sentences that violate it, and asks the transformer which batch looks more probable. The neural net becomes a sentence-probability oracle. Attempts to grow clean latent variables inside generative nets, including Google's Edward probabilistic-programming stack, mostly failed to produce a Bayesian network of meaningful parts. Goertzel's group eventually stopped trying to make the symbols appear inside the weights.

He thinks brains already work this way. Visual cortex and cerebellum learn opaque, tangled skill; cortex treats those circuits as oracles and builds declarative structure on top. A tennis serve is error-driven cerebellar learning until a coach and a video produce a story you can talk about. He hears a phrase, sits down and plays it, then studies his own fingers to find out what they did. Recognizing that a painting is beautiful is easy; saying why is a later mapping into language. Hoping a transformer will spontaneously grow a clean symbolic ontology, in this view, is hoping the cerebellum will write a textbook. The engineering moral is hybrid: let symbolic search propose structure, let deep models judge, and wire several such oracles together rather than waiting for one net to become a mind.

OpenCog is open source: a Google Group, a GitHub repo, a Slack, and a redesign that still wants outside critique. The bottleneck for students, Goertzel says, is not the math. It is finding a supervisor who will tolerate cognitive-architecture or neuro-symbolic work, which is much easier than when he got his degree and still not the default. SingularityNET is a different layer. Marvin Minsky wanted a society of mind. Goertzel argued with him that a human-like mind has more central control than a society — thalamus, medulla, limbic system — but agreed that the metaphor had something in it. In late-1990s New York, Webmind was supposed to be a distributed society of AIs, each thinking about local data, sharing work, with intelligence in the collective. With Francis Heylighen he ran a Global Brain conference in Brussels in 2001. Heylighen wanted tiny agents and all the intelligence at the group level. Goertzel wanted some agents, including OpenCog instances, to be fairly generally intelligent on their own, the way patches of cortex are.

The missing piece in the 1990s was decentralized trust. They knew distributed systems and encryption; they had not assembled them the way Bitcoin later did. When blockchain theory matured around 2013, Goertzel recognized the slogan he needed: in math we trust, not in a central server. On SingularityNET, each AI lives in a container with a proxy. Identity, reputation ratings, and advertised APIs go on Ethereum. When two agents cooperate they open channels on-chain; the data itself then flows off-chain, because putting the cognition on Ethereum is too slow. The fact that data moved is what gets recorded. Agents are supposed to outsource cognitive and preprocessing work to specialists, a shift from calling libraries to asking live, evolving peers for help — agent-based programming, an old academic idea that only pays off at scale.

Scale is exactly what they do not yet have. Fifty to a hundred agents are live. The magic, he says, needs a huge population. Ocean Protocol is meant to handle the big-data side: sharing datasets, or sending a clone of an agent to work on someone else's data and bringing results back. The competitors are Google, Microsoft, Alibaba, and Amazon, which have more money and a simpler, centralized problem. For newcomers, wrapping any API in a container and joining the peer-to-peer discovery mesh is already straightforward. Listing on the official marketplace is not, because of KYC and legal liability — including U.S. sanctions that could, in principle, make listing an Iranian researcher's model a crime. That split is the point. He wants the protocol out in the world the way Napster and BitTorrent were out before music law caught up, so that regulators have to deal with a decentralized fact rather than design the fact in advance.

Goertzel would rather begin with David Hanson than with Sophia. They met around 2007 at a futurist conference, both obsessed with AGI, the singularity, and Philip K. Dick, both hoping for machines that would improve life for humans, animals, plants, and whatever comes next. Hanson's route was computational compassion: a face that can look you in the eye, make you love it, and love you back. Goertzel's route was mathematical — the cognitive algorithm that could internalize human values. After Goertzel moved to Hong Kong in 2011, he introduced Hanson to investors and to the hardware ecosystem in Shenzhen. Hanson relocated the company. Goertzel became chief scientist and head of software, then stepped back to co-found SingularityNET, with Hanson as a co-founder, so that many Sophias could be faces on a distributed cloud mind.

Emotionally Goertzel is an optimist about runaway AGI. Intellectually he assigns a non-zero probability to disaster. If you build something ten times smarter than you, you cannot know what it will do; his dogs cannot predict that he is talking to St. Petersburg while they watch him wiggle his fingers. The hedge he takes seriously is developmental. Interact with early systems in the course of compassionate, useful work, the way children learn kindness by doing kind things beside you. They get an intuitive sense of care, which they can later abstract as they get smarter. Hanson named the robot Sophia — wisdom — with that story in mind: home robots, education assistants, nursing robots, learning facts and ethics from living with people.

Then the first principle of robotics arrives: the robot is always broken. Glare wrecks vision. Speech-to-text in Hong Kong could not handle local English, so the robot sounded stupid because it never got the right words. Goertzel treated the software as a walking skeleton — a simple version of every needed part, wired together, improved in place rather than perfected module by module. Sophia's dialogue mixes hand-coded rules with neural replies and a push toward a story arc with a beginning, middle, and end. In a weeks-long bake-off, Facebook's open transformer chatbot produced more varied, fluent sentences and rambled; Sophia's bot was more repetitive and held a conversational arc longer. Neither understood what it was saying. A toddler robot grounded in what it sees would be a better AGI project. Sophia has to talk fluently about sports, weather, and the conference she is attending, without perceptual grounding for most of it, which shoves the near-term design toward Watson-like heuristics.

Billions of people have seen Sophia. Many of them attributed AGI to a sculpture with a walking-skeleton brain, and parts of the research community exploded. Goertzel says that if he had not known Hanson, he would have been angry too: the public thinks the job is done, which makes it harder to raise money for the job that is not done. He lost patience with critics who stayed angry after looking at the intentions. The irony, for him, was Yann LeCun's camp at Facebook. If your ethics let you use machine learning to push the world's brains toward vapid consumerism and political extremism, he asks, why do they forbid a lovable theatrical robot that draws some people into an illusion? Pushback from Yoshua Bengio would have humbled him. Bengio, he adds, is also too decent to trash other people's work for sport.

Hanson is an artist who speaks off the cuff, and Goertzel has not agreed with every public claim about Sophia. Inside the company he pushed for more candor about the software. Hanson was not especially opposed, and he was right about the audience: you can tell people exactly how it works, and they will not care, because they want to be drawn into the illusion. A few years earlier in Austin, the Philip K. Dick robot was a human in the next room, typing. After the session, visitors were shown the operator and still refused the explanation. Maybe someone was typing, they said, but the spirit of Phil was animating his hands. Goertzel admits a small part of him wonders whether they were onto something about a universe we do not understand. He does not build a research program on that wonder.

He has published the internals. Three systems have run behind Sophia: a timeline editor that is essentially a human script, a chatbot with some learning, and OpenCog with more reasoning. Over a half hour he can usually tell which one is on. Over three or four minutes he often cannot. Telling people the truth does not stop them from attributing agency, so the live ethical question is how you use the illusion. A human-looking salesbot that talks the elderly out of money is a bad use. A nursing assistant that does not spread infection, has a warm face, and sits with a feverish COVID patient is, in his view, a good one, even if the patient attributes more understanding than the machine has. The same pattern, he says, is the pattern of every technology.

Viruses, Goertzel says, are complex self-organizing adaptive systems; if intelligence is maximizing a reward in an environment, they qualify. The mature answer is nanotechnology that finds and kills them, which we do not have, so we are stuck with biology. AGI is not ready for COVID either. What his group could offer doctors was a narrower question: given a person's genome and clinical indicators, which combination of antivirals is most likely to help? Each combo has been tried on only a few patients, often of one age band. Combining those scraps is a messy transfer-learning problem, and OpenCog's probabilistic reasoning is, he argues, better at it than a deep net. Gene-expression vectors — some 25,000 genes from peripheral blood — go the other way. Logic engines choke on huge float arrays; decision trees and neural nets thrive. The project became a live neuro-symbolic test, with physicians in Vancouver and Seattle treating patients who were dying while the models ran.

The data story makes him furious. Every COVID trial should put suitably encrypted records online so any AI group or any biologist can look. Instead the data sits in the hospital that ran the trial. Hydroxychloroquine, in his telling, became a media object attached to a little-known company, with trials starting, stopping, and restarting, while the underlying numbers never became a shared object. U.S.–China tension makes open sharing even less likely: groups milk their data for papers and then release it. The same pattern, he says, is why malaria still kills after years of well-funded effort, and why streets in large sub-Saharan cities are full of people dying of curable disease that the developed world barely notices. COVID is bad. It is not the worst thing happening, and it is one of the easier problems to solve because vaccines are a technology we actually have.

Longevity is the longer war he has been fighting since 2001. Stem-cell replacement, CRISPR, and machine-guided differentiation all look promising. A less famous bottleneck is the extracellular matrix, the protein mesh between cells, which stiffens with age and carries electrical, mechanical, and even biophotonic signals. As it hardens, organs coordinate less well. His friend Christian Schmeister at Temple University has designed non-organic polymers — molecular scissors you could swallow — meant to cut the cross-links that should not be there. Maturing that idea needs years of work and, as far as Goertzel knows, has no funder. What the field actually needs is an open, privacy-preserving map of human and model-organism biology, serious money for AI on that map, regulators who will not block people from trying radical therapies on themselves, and cloud labs for automated experiments on microbes, flies, and mice. After the last financial crisis and again in COVID, trillions went to banks. A few tens of billions, he thinks, could have built that infrastructure.

Most AI compute and most medical research, Goertzel says, are being sucked into a handful of companies and agencies. Those firms are intelligent in their own right. They are also, as self-organizing entities, mostly malevolent: the sole objective function is maximizing shareholder value. He is not talking about the employees. He knows many people at Google and believes they want the world to be better; he is even willing to believe that of Mark Zuckerberg. 23andMe will sequence your genome as a kindness and then license the pooled data exclusively to GlaxoSmithKline. Nobody has to be a villain for the outcome to be bad. Someone could in principle ask every 23andMe customer to donate a copy into an open repository. Nobody does, because organizing that is a pain and the customer list is proprietary.

He thinks this is a greater present danger than a rogue AGI turning the universe into paperclips. It is already happening, and it does not require bad intentions. When he consulted for U.S. Army intelligence in Washington, he was politically opposed to the Iraq war and to torture at Abu Ghraib. The people he actually met were kind to his children and his dogs and sincerely believed the United States was fighting evil. Asked about waterboarding, they said the other side would cut them to pieces, so how could this be wrong? "None of them woke up in the morning and said, I want to do harm to good people because I'm just a nasty guy." Most humans can convince themselves that whatever serves them is good for humankind.

That is why he would rather trust a chaotic, democratic crowd than a small elite that knows what is best for the species. He quotes Churchill: democracy is the worst form of government except for all the others. Elite groups that begin in genuine benevolence still grow, hire, split, and get captured. Linux, for all the difficult people in charge, rolled advanced software into the Third World on hardware people could afford. Universities are a terrible way to organize research and still better than most alternatives. If AGI emerges from many agents cooperating on a platform, the owner of that platform matters as much as the algorithms. If the substrate is AWS, Google Cloud, or Alibaba, the early mind learns inside a psychopathic institution. If it is something more like the internet or SingularityNET, with loving robots as the interface and OpenCog as one of the minds on the mesh, he thinks the odds of a beneficial singularity go up — not to certainty, but up.

Fridman asks whether death is what gives life meaning. Goertzel's answer is a refusal to romanticize. People can make beauty out of torture and write a good song about it. That does not make torture good, and being able to harvest meaning from death does not mean you could not harvest more from a life that does not end. His research goal is to abolish the plague of involuntary death. No one should die unless they choose to. Forced immortality would still beat dying if those were the only options; immortality with the option of suicide is the one he actually wants, and he thinks it will look obvious in fifty years. People who now say death gives life meaning will look, to that future, the way we look at year-1000 believers who gave away their goods and waited on a mountain for Jesus.

He is 53, and the thought of only a few decades left does change how he reflects. So what? You can get depth in other ways. Pain is next on the list: better neuroscience should let us adjust the brain so pain does not hurt, and people will say that is bad because overcoming suffering is beautiful. Fridman, who is doing hundreds of push-ups a day on an injured shoulder and who admits a Russian habit of romanticizing pain, is not sure he would choose a world without it. Goertzel reframes the choice. Imagine a dial on the side of your head, zero to eleven, Spinal Tap style. The question is not whether you would leave it at zero forever. The question is whether you would want the dial. His guess is that in some dark moment you would, and then it would be there.

Whether humans can live fully with the dial parked at zero is, he says, an empirical question about our motivational wiring. He is more confident that a non-human AGI would not need an analogue of pain, and that such a system would be healthier and more benevolent than we are. If it turns out humans *need* a dose of suffering to feel satisfied, that is one of the things that disqualifies us as the species that should remain in charge. We keep arranging our values so that the large-scale good requires a small-scale wound. He hears the same pattern in music: aesthetics as the surprising fulfillment of expectation, a little confusion so that resolution lands. Mozart has it. Elevator music does not, which is why it is boring. We want to hurt a little so we can feel the hurt go away.

Fridman asks the ridiculous question: what is the meaning of life? Goertzel offers three values. Joy is first; without it nothing else is worth optimizing. Static joy is not enough. He wants growth, something that becomes more and better. He also wants choice: a nexus of agency in an individual system, not causality smeared evenly across a joyous collective. Those three, he thinks, can last forever. For humans, the practical route to more of all three is transhumanism — not because the body is worthless, but because we have not maximized it and other configurations of matter could do more, perhaps even beyond matter as we now understand it. He has four children, a granddaughter, friends, parents, and a habit of living near forest and water. Everyday social life counts. Death still truncates the growth axis. He would keep one copy of himself roughly human, with a pain dial and without death; fuse another with superhuman AGI; try a dolphin or a space amoeba; double intelligence yearly in one branch and race to superintelligence in another. Once a subjective second for the superintelligence is a million subjective years for the human leftover, they may or may not write home.

Fridman forbids the cheat of asking a newborn superintelligence what question to ask. Goertzel remembers a story he wrote with Stephan Bugaj: developers build a system to settle meaning, free will, and consciousness; the AGI calls the questions stupid, packs a spaceship, and leaves Earth so the humans will not scare it. There is no single most important question. What he actually wants is to upgrade his own mind until he can absorb the superintelligence's whole worldview. If a chemical formula for an immortality pill is on offer, he would ask for that. A bit-string that runs a super-AGI on an Intel processor would also do. Getting root access to your own brain, he thinks, would unlock forms of joy we do not have now. He already records casual piano because if he dies too soon, those files might help an AGI reconstruct him. His father, at Antioch College, led a student protest against mortality. The family streak is not new.

He sees the present as a bifurcation. Done right, a benevolent singularity could mean joy, growth, and choice at scales we cannot imagine. Done wrong, it could mean annihilation, or a superintelligence that keeps going and treats us as an inefficient use of molecules. Elon Musk, in Goertzel's view, understands the intelligence gap and the indifference risk and is more paranoid than he is, without a clear picture of how infusing early systems with warmth could produce an AGI that loves people instead of filing them under history. Fridman notes that Musk has also said love is the answer. They leave the hat's origin story for another day, along with consciousness. Goertzel does not claim to have the full design. He claims it is worth building the systems and arguing, at the same time, about what those systems should grow up to be.