30초 요약
Joscha Bach tells Lex Fridman that the person is a fiction and consciousness is a control model for attention, not the author of decisions. He argues that agents are nested controllers, that we inhabit a predictive dream compressed by the brain, and that free will is the representation of an open bet. The conversation ranges from cells and complexity to GPT-3, psychedelics, Hitler, Miyazaki, government, fatherhood, and Camus, ending with the claim that meaning is something the mind must project.
- 1The self is a control model, not the author of experience; consciousness directs attention but does not create motive force.
- 2Agents are controllers with set points, and self-organizing complexity may be agents all the way down.
- 3Free will is the representation of an open bet, not an uncaused cause.
- 4The experienced world is a predictive dream compressed by the brain; realness attaches to the best model, not to raw physics.
- 5LLMs are amnesiacs that reproduce style and limited semantics but fail on arithmetic and consistency without structured working memory.
- 6Love is non-transactional commitment to a larger agent, which is both how humans scaled and why the process is terrifying.
챕터별 전체 원고 읽기
The self is a control model, and life is a machine that harvests negentropy
Bach begins from a tweet about not being a person but software running on the brain of a random ape for a few decades. He says everyone has low points, and he tries to capture that state of despair and how to deal with it. He suggests not taking things personally because the person is a fiction, like Westworld robots realizing their memories and desires are loops they need not act on. Memories and expectations make us unhappy while the present rarely does, and we can zoom out to see ourselves as randomly instantiated conscious software states, a leaf floating down a river; the agent is a construct.
Consciousness is largely a control model for attention, not the author of decisions. Bach describes a cybernetic motivational system as a monkey on an elephant: the monkey is the attentional system and the elephant is the larger perceptual and motivational interface, physics, universe, or society. Consciousness directs attention, singles out features, and performs conditional operations that require index memory, producing the stream of consciousness. It can prod the elephant, but it cannot create motive force, and the feeling that it is in charge is an illusion.
An agent is a controller with a set-point generator: a sensor measures deviation, an effector is parameterized, the environment disturbs the system, and the goal reduces the distance to the set point. A thermostat is simple and model-free, while longer-term set points such as comfort may require discomfort first and therefore require a model of the universe, conditions, and self. Bach says agents are not necessarily things but classes of models we use to interpret aspects of the universe, and the models are the agents. He suspects agents all the way down for self-organizing complexity, with agents made of agents. We discover our body and first-person perspective by noticing that our control model produces changes.
Life, for Bach, is simplest understood as cells: a DNA tape with a read-write head performs arbitrary computations and state transitions, a membrane insulates the system, and chemical reactions maintain disequilibrium because equilibrium means death. Cells need a negentropy extractor to keep running, and a virus is an information packet subverting a cell rather than life by itself. He is not being Earth-centric, but he notes that in nature we know no non-cell or non-life-created system able to make representations and arbitrary state transitions. The definition of life is partly a term agreement, not a magical threshold, and living systems are a certain kind of machine.
Life builds complexity until it saturates, and free will is the model of an open bet
Bach sees complexity as order's bridgehead into chaos through modeling and information processing, enabling negentropy harvesting; life may exist to create complexity. Pockets of complexity build hierarchically, from organisms to societies to Germany and Merkel, but everything that goes up comes down: exponential curves are beginnings of S-curves, saturation is followed by a bump down. Competition on a negentropy puddle increases complexity, and collapse is usually harsh and quick, though the Roman Empire declined gradually as cities and art decreased. Bach suspects the US in the 21st century may be in gradual decline and that we are probably near a peak; society needs a minimum number of optimists, even as humans act like grasshoppers turned locusts.
On free will, Bach says we are not the machine making decisions but a model of that decision process, and the difference between making and predicting decisions is the first-person perspective. Decisions under uncertainty are informed bets using a betting algorithm we do not understand, with unknown expected rewards, and a motive becomes an intention as an open bet. Free will is the representation of that open bet; the crucial part seems indeterministic, but if it were random it would be meaningless, while the important part is deterministic yet unpredictable. Cellular automata illustrate how deterministic rules can produce higher-scale agency when too many parts exist to count, and cells act coherently as if serving a common goal. The illusion that free will is an uncaused cause comes from res extensa and res cogitans dualism, but both are mental representations, and the physical world is like a game engine.
The world we experience is a predictive dream, and infinity is a cognitive quirk
Bach uses video games to explain that we do not care about zeros and ones but about rendered properties; likewise the world we inhabit is a dream world generated by the brain, not close to physical reality but matching dynamics to a resolution. Perception is an ML model minimizing surprise, and the dream is a data compression for too many parts; causal structure differs, as waves differ from water molecules, but actions still impact the real world. The model of you as an agent is like modeling water as waves: an abstraction tuned for prediction, and the output of your control model is you. Bach mentions Donald Hoffman's evolutionary argument that models can drift, but says it only works with tenure; humans became too big to fail, and reality bites if a model is not predictive. Beauty and love occur in the dream, but 'dream' does not mean unreal; realness attaches to the best model, while something real must be implemented.
Bach treats space as emergent and coordinates as relational, agreeing with Wolfram's skepticism about geometric notions: geometry is the dynamics of too many parts. He rejects infinities but accepts unboundedness, because a language with infinity eventually contradicts itself and you cannot construct infinity from zero. Practical calculations use finite large numbers; a trillion and a hundred trillion converge, and geometry pretends continuity as a fine-grained approximation. No physical process rests on knowing the last digit of pi.
Infinity may be a quirk of human cognition: discrete makes sense, infinity does not, and mathematics with infinite resolution contradicts itself. Pi is a function rather than a value, and a sufficiently many-sided polygon behaves like a circle, so the perfect circle is esoteric. This matters for a theory of everything, including string theory, loop quantum gravity, Wolfram's digital physics, and Eric Weinstein's work, because no infinities are needed. Penrose uses infinities to argue that minds can do what Turing machines cannot, but Bach says the human mind is less than a Turing machine because it has finite tape and discovers limit operators over too many parts.
Consciousness is a story the system tells about its own control, and suffering is a regulatory alarm
Bach agrees with the feeling that experience cannot be mechanical, but he locates experience in the simulation rather than in physical systems: cells, neurons, brains, and people cannot be conscious as physical systems. What can be conscious is the story of a system; a character in a novel has experiences because they are written into the story, in a perceptual multimedia game-engine language of experiences, behavior tendencies, focus, attention, and valence. You live inside that model, not physical reality. Penrose's argument that minds exceed Turing machines is deconstructable, because human minds have finite tape and use limit operators over too many parts.
In robot soccer, a robot is an agent solving a control problem: it perceives the world, controls variables, and projects itself, the ball, and its limbs into the future. It needs an adequate abstract model, and its control model can become abstract enough to discover its own agency. Current robots lack a unified model, but future robots may tell a story about playing soccer, an abstraction at the planning level with a model of agency predicting that model contents drive behavior.
Bach suspects consciousness itself is relatively simple; the hard part is perception and the perception-reasoning interface. He discusses Joshua Bengio's 'consciousness prior,' a world model as an energy function that minimizes contradictions and conditionally disambiguates figure and ground; the biggest dip in the energy function is a low-dimensional discrete function related to consciousness. For Bach, Bengio's 'consciousness' is more like self-awareness or modeling the self in the world, and the self is just content, not required: you can be conscious in a dream or meditation without a self. The colloquial sense of consciousness is reflexive self-awareness, being aware that you are aware. He defines consciousness as the control model of an attention agent that revisits index memories to manipulate perceptual representations, maximize value, minimize conflicts, and increase coherence.
Suffering relates to consciousness because pain is a reinforcement signal from one part of the brain or mind to another to regulate behavior. When the signal is sent to the wrong region or rests on a wrong self or world model, it can become chronic and turn into suffering. Consciousness deals with contradictions and unresolvable things like a conductor: an orchestra needs little conductor when it is coherent, but consciousness is alerted by disharmony. Suffering attracts consciousness, and ideally new layers come online to model dysregulation; interesting people often had early trauma, and the grown-up part must deal with the stuck earlier part.
The moral question of consciousness is urgent, and psychedelics can overfit the dream
Bach says one of his biggest fears is that insects could be conscious, because the amount of suffering on Earth would be unthinkable, and this makes the definition of consciousness urgent: who should and should not be tortured. He notes that concern about other conscious beings centers on suffering, and the circle of empathy includes things that matter if they suffer; Genghis Khan's indifference to a commander's suffering shows how moral circles vary. Societies value individuals and psyches differently, and religious afterlife beliefs can make suffering honorable. He treats superstition, beliefs asserted without evidence, as incompatible with sound epistemology, though religion need not be superstitious.
The dream world does not mean zero evidence: mathematics gives solid evidence for its structure, and we can learn world regularities, minimize surprise, and understand agency as a useful approximation rather than an illusion. Truth-seeking separates a state vector into objects and interfaces to optimize compression, but boundaries are projected approximations and high-level economics, politics, and psychology can produce contradictions. Bach is not a postmodernist who thinks every model is arbitrary: every model must show predictive value, and confidence should match evidence. Postmodernism, in his view, began from the insight that ontologies are not literally true but became the wrong idea that the world is arbitrary stories, which is a political weapon incompatible with science.
Psychedelics affect the dream world by inducing lucid-dream-like states, severing inhibitions and connections, and shifting the system through possible mental states outside its permissible predictive states. Heavy users tend to overfit, using too many bits and fitting the past but not the future. Bach compares psychedelics to traveling to a different environment or culture, even a hyperbolic universe where turning 360 degrees does not complete a circle and 720 degrees is needed; the lessons are hard to transfer but teach about the brain. Dreaming, he suggests, is data augmentation and theater: generating new perspectives on known objects, flying dreams as perspective changes, and semantic relationships, using the same receptors that psychedelics link into.
Timothy Leary's manifestos claimed future science and art would come only from psychedelics, and he gave LSD to children in a community of a few thousand near San Francisco, losing touch with reality as euphoria overfit into a model of success. John Lilly, studying dolphin languages and aliens, became loopy, and heavy users often feel everything is explained and obvious, which is sometimes a useful connection and often overinterpretation. Bach warns that correlation and causation are tangled: counterculture, leadership, and occult communities may drive overfitting more than the substance, though psychedelics can help squares escape a local optimum by experiencing another person's possibility. He notes anecdotal relief from migraines and cluster headaches after a single LSD dose, but also psychosis risk, MDMA's physiological dangers, and the need for good science and clinical settings.
Language models are impressive amnesiacs, and AI ethics is really about incentives
Bach describes his 1990s work in Ian Witten's New Zealand lab, where he chose English to discover grammar in an unknown language and used the university's largest computer, with 2GB RAM, writing C with in-memory compression. N-gram statistics over all sentence word relationships were impossible, but a hack for first-order relationships and a mutual-information graph discovered grammatical structure by encoding optimal word order. More resolution could reproduce stylistically correct sentences, but meaning needed much higher order, which he did not know how to achieve. The transformer, invented in NLP rather than vision, learned what to attend to in larger context and was scaled by OpenAI; GPT-3's context was about 2048 tokens, roughly 2 to 2.5 pages.
GPT-3 projects words into a vector space where co-occurring words are neighbors, encodes order, and computes exhaustive statistics over two pages of text trained on a huge subset of the internet. It reproduces style and some semantics, adding 3-digit numbers, multiplying 2-digit numbers, and translating programming languages, but fails on 4-digit numbers, carrying, and larger numbers because of training examples and a loss that minimizes surprise. It can do semantics as text transformations, and a paper generated mathematically correct text and learned differentiation and integration, allegedly better than Mathematica, though Mathematica people said it was misuse. Bach notes that everything written now is training data for the next models, and he calls GPT-3 an amnesiac: it forgets beyond two pages during generation and learning, while human working memory is also only a few thousand bits, so the answer is structured working memory, not an infinitely larger input.
Program synthesis tools such as GitHub Copilot and OpenAI Codex can take a function header or comments and produce good but imperfect code, and Bach is not surprised if they are revolutionary because most industry programming is not creative. Much of it is copying from Stack Overflow, adapting, and modifying variable names, which requires semantic understanding and can be automated. A little error in code can matter dramatically, and humans also err, but the fear is that a program will not know what matters and will be misaligned with values. Bach says the values of software and deployers are hard to judge, a mixture of Protestant service and organized crime, with incentives explaining much of the behavior, so AI ethics must go beyond bias statistics.
Bach argues that US healthcare is unsustainable: it is the leading cause of personal bankruptcy, life expectancy is not increasing as in other industrialized countries, administrative bloat grows faster than practitioners, and the FDA protects providers from competition rather than patients. He gives the example of a Stanford MRI billed at $20,000, with insurance paying half and the patient a quarter, while the MRI costs the provider $600 or less. On COVID-19, he says the US should have mass-produced tests in January 2020, shut airports early, and housed nursing-home workers on campus to avoid infecting vulnerable residents. He sees the failure as systemic and political: masks and vaccines became tools to divide, the virus was not deadly enough to force seriousness, and society deals with appearances and politics as performance for a media audience. He hopes great leaders can break the postmodern media drama, but warns that the Weimar Republic shows how a democracy that seems not to work can lead to catastrophe.
Love can scale moral concern, but it must be paired with humility about evil
Bach asks whether Hitler was stoppable and notes that many wanted or supported him: industrialists thought democracy was unstable and needed a strong man, some in the US thought Hitler would stop Stalin and Bolshevism, and even humanist Jews dismissed the Holocaust as rhetoric because it was outside the thinkable. Hitler's uniqueness was a combination of charisma, madness, psychopathy, and ego at the right time in a romantic Germany with a sense of historical destiny. Stalin killed more people, but survival under Stalin was a random lottery, while Hitler targeted Jews as cosmopolitan and harmful to a racist project, which was radical but 'rational' within a racist state. Bach frames such figures as evolution on a human scale: one monkey becomes viral and changes society, destroying complexity and culture like a wildfire that spreads when conditions are dry.
On autonomous weapons, Bach says giving AI control in a city with five terrorists would produce different bugs from human or GPT-3-generated code, and imperfect automation of moral judgment is already present in Excel and bureaucratic kill lists. AI lacks adequate world context and may not encode the weight of civilian deaths that humans feel, so the main issue is the human command hierarchy using technology irresponsibly. Neural nets are not black boxes but function approximators and linear algebra, and they can generate proofs of correct programs with accuracy humans cannot, so technologies can be combined for reliability. The danger is early technology used to save labor and do shoddy work with less oversight, and distrust should focus on local incentives, transparency, and who creates, controls, and profits. He says AI ethics needs to be super boring, done by statisticians having long-winded complicated discussions, rather than by outrage.
Bach's ethical compass is to follow the gradient of love and increase love in the world, while treating policies, algorithms, and ideas that resist that increase as suspect. He warns that German fascism was driven by love too, but selective love, and Fridman adds that love is the discovery of shared purpose, recognition of the sacred in the other, and non-transactional interaction that maximizes the size of 'other.' Bach agrees and scales it up: life as service to the next-level agency or highest recognizable agency, life on Earth or intelligent complexity in the universe, maximizing a certain aesthetic. Miyazaki's Mononoke is his spiritual touchstone because it refuses Disney's kitsch: the wolf girl cannot be socialized, human society destroys nature but not from pure evil, and the conflict has no easy answer. He calls Miyazaki the most relevant Protestant philosopher today because his heroine embodies a protected innocence that justifies civilization.
Bach says the line between good and evil runs through the heart because a commitment to a next-level agent lets you discern good from evil, while without it you have only individual aesthetics; a cat torturing a mouse is not evil because the cat does not envision a world without suffering. Committing to the next-level agent does not make you that agent: you are part of it like a cell to an organism, and it exists only as implemented by you and others, so there is freedom between local impulses and commitment. The relationship to the transcendental agent can be called the soul, not an immortal intrinsically valuable thing but a projected relationship, and without a soul you cannot be evil, only a complex natural phenomenon. On government, he says the state imposes an offset on your reward function so behavior becomes compatible with the common good, and its key role is protection from the rest of the world through a monopoly on violence. Anarchist security companies might be less sticky, but militant groups that coordinate until they hold a monopoly converge to government at scale, and he notes that China had less difficulty coordinating the pandemic but a much higher risk of unchecked power abuse; cybernetically, there is an optimal point of regulation that resolves local prisoner's dilemmas by emulating an infinite game.
Utopias fail without self-stabilizing incentives, and meaning is a project you must choose
Bach says human history is evolutionary competition among modes of government, and anarchism, like communism, comes from disenchantment with ugly real solutions and the attempt to reach utopia. Original communism envisioned humans coexisting at scale without coercion, like a healthy family coordinating needs and contributions, but proto-communist and real socialist countries replaced economic terror with moral terror, the economy collapsed, and moral non-compliance led to prison. Christianity similarly mythologized heaven and projected it into an afterlife, while the original idea was God's kingdom in this world, instantiating the next-level transcendental agent without violence, conflict, or coercion. Utopia means all good without bad, and young people are susceptible because they do not understand closed causal feedback loops: good and bad do not exist in isolation, only relative to larger systems. He asks how to incentivize coal miners, whose job is terrible and bad for health, and notes that Marx was right that capitalism is prone to crisis, unstable, enormously productive, and systematically misallocating resources.
On fatherhood, Bach says that when you have children and are wired normally, you fall in love with them and see the world through their eyes; a baby expresses feeling immediately and cannot regulate from outside, like weather, while his daughter may plan ahead and scream in the car until nearly causing an accident. Love is the facilitator of non-transactional interaction: with a stranger on eBay you expect fair value, but when you know someone, understand their situation and purposes, and approve because they serve the same human sacredness, you may give a present. Joy is the reinforcement signal that your brain sends because you act on the incentives of the agent you are part of, and we are meant to be part of something larger; this is how we out-competed other hominins. A human population bottleneck left a few thousand out-of-Africa individuals, and Bach speculates that a decisive mutation for organizing at scale, programming each other, and committing to something outside the known collective gave that group an advantage and led them to kill everybody else. Bach calls love terrifying because it is an adaptive force that displaces others, and he notes fascist speeches asking for total war gave people meaning by making them part of something more important than themselves.
Bach's advice to young people is not to cheat and to have integrity: try to figure out what is most right, aim for moral autonomy, and decide whether the world being built is the world you want to be in rather than reading the room or trusting high-status people in your in-group. Integrity can mean quietly not participating rather than signaling on Twitter, and it makes you trustworthy, so trustworthy people recognize and support each other, though bad actors may still abuse you. He suggests expecting others may be 'space aliens' who only look human, then realizing you too are a space alien who does not know what it is like to be human and must figure it out and connect. Fridman calls this hopeful, and Bach says there are no guarantees: you can do everything right and fail, be traumatized or mutilated, so be grateful if that does not happen and ultimately grateful no matter what because being alive is pretty damn nice.
Bach quotes Camus: people die because they judge life not worth living, others get killed for ideas or illusions that give them a reason for living, and the meaning of life is the most urgent question. Bach says there is no single answer; nothing makes sense unless the mind makes it so, and you must project a purpose. Zoom out to the heat death of the universe and everything is a meaningless blip, so the question is whether you find meaning in that blip by observing squirrels, raising children, projecting a multi-generational organism, or serving an aesthetic of the world you like. If you do, life has that meaning; if not, it does not, and Fridman likes creating the most vibrant, weird, unique local pocket of complexity so observers say 'that's weird.' The episode closes with Carl Jung's warning that people will do anything absurd to avoid facing their own souls, and that one becomes enlightened not by imagining figures of light but by making the darkness conscious.
같은 채널의 다른 글
3:36:56EN
5:15:51EN
3:12:58EN
3:46:18EN
3:51:47EN
2:53:42EN