BCI Would Take 30 Years. This Studio Pivoted to Alignment Instead

Eliezer Yudkowsky has a reputation in alignment circles for calling other people’s research agendas stupid. In 2024 he left a different note on an experiment design from AE Studio: “Not obviously stupid on a very quick skim. I rarely give any review this positive. Congrats.” Nathan Labenz, hosting The Cognitive Revolution, told listeners he was warmer than that. Both of the studio’s recent papers, he said, ranked among his favorites of the year.

The guests were Judd Rosenblatt, AE Studio’s CEO, and Mike Vaiana, R&D director on the alignment team. The company is a bootstrapped software consultancy of more than 160 people. It takes no investor money, writes code for clients, and pours the profits into research. From the decision to shift into alignment to two public results, the turnaround took a little over a year.

One result is self-modeling: add a “look in the mirror” term to the loss so a network predicts its own internal activations, and the internals simplify while task accuracy holds. The other is self-other overlap: take an agent that has already learned to lie, fine-tune it with a tiny extra objective, and the lying disappears while the work still gets done. Labenz stressed the shared selling point. Both strategies are relatively straightforward and understandable. They do not require breakthroughs in interpretability, or in any other branch of AI safety, in order to work.

Rosenblatt dated the company to 2016, when he realized he could live off his wife’s salary and never work again. That luxury, he said, freed him to think about the long-term future of technology. He liked long, complicated thoughts that took a while to turn into words, and by the time the words arrived they were often not what he had meant. Humans, he argued, are bad at communicating even to their future selves. If brain-computer interfaces became the main way people used computers in his lifetime, he wanted that interface to increase the user’s agency rather than manipulate them, interrupt them mid-thought with ads, or let Mark Zuckerberg own the extensions of their thoughts. He also thought human augmentation might be needed to solve alignment at all.

The plan had four steps: start a profitable software consultancy with no investors; use the profits to incubate agency-increasing products; put the capital into BCI software without a profit motive; and, eventually, make humans wise enough to solve alignment. Labenz said that if the plan had shown up in a pitch competition he would have objected that there were way too many steps. Eight years on, the first three were done. The consultancy grew past 160 people on a single guarantee: treat the client’s project as if it were AE’s own startup. Everyone primes themselves with that line daily, so the top idea in mind in the shower or walking the dog is how to make the client succeed. The office was listed on Airbnb in the early days. On a partner’s first day, a Dutch tourist stepped out in a towel and asked where the bathroom was.

The first self-built product was an SMS tool for subscription e-commerce. The perennial problem in that business is that people order coffee or protein bars, eventually have too much, and cancel. AE’s tool let customers change quantities or skip a month by text, with no promotional pushing. Short-term revenue for the merchants fell; lifetime value rose. Rosenblatt said that if the company had been bigger, he would have acquired the firm that acquired them and used that reaction as a litmus test for every subscription brand. The underlying ethic came from Paul Graham’s essay “Mean People Fail”: the world is moving from taking resources to creating value for others, and the founders Graham respects are people others want to work with again.

Culture inside the firm has a name, “good tripper,” from summers Rosenblatt spent leading wilderness trips in Wisconsin and Ontario. A good tripper carries more than a fair share, sets up tents, chops vegetables, and purifies water without being asked, and pushes a canoe farther than seems possible. “There’s no better feeling,” he said, “than throwing the canoe off your shoulders into the next lake.” His wife, the company’s CTO, set up a rule that 5% of profits go to effective-altruism charities, often malaria nets, so that every hour spent on an annoying client bug also donates toward saving lives. Internally they also built Universal Launcher, a capture tool Rosenblatt credits with making him a hundred times more prolific: one shortcut dumps a thought into an external system he trusts will execute, which frees working memory. Vaiana said that when an install broke it, he felt he could no longer use a computer. The learning curve is steep, like an instrument, and the payoff is that the next distraction no longer sits in your head all day.

The BCI work was real, not a slogan. AE won the Neural Latents Benchmark challenge started by Chethan Pandarinath’s lab at Georgia Tech. They shipped an open-source neural data simulator, built with a collaborator named Chad, so developers could iterate without a human participant in the loop. They contributed a neurotech development kit, data-sharing standards around Neurodata Without Borders and BIDS, and privacy-preserving methods using federated learning and homomorphic encryption. Collaborators later included BrainGate, Blackrock Neurotech, which makes the Utah array, the first FDA-approved implanted electrodes, and Forest Neurotech, whose minimally invasive ultrasound systems pull about a thousand times more data from the brain than earlier approaches. Rosenblatt admitted the first forays were embarrassing. They still made a name.

Labenz asked whether the point was the one Elon Musk talks about: raising bandwidth so humans can keep up with AI. Rosenblatt was blunt. The original motive was to make humans smart enough to solve alignment in the first place. Keeping up with AI via an implant, he said, is a low-probability, somewhat wishful idea. A few years ago he had kids, watched AI speed up, and the company ran the numbers. Internally they concluded that getting BCI to the point where it would substantially accelerate alignment was maybe at least 30 years out. If AGI is a few years away, 30 years cannot help. They still want a portfolio of bets across timelines, because people update too hard on the latest news, the way they update too hard on a presidential race, and long-horizon work dies when everyone abandons it.

They used effective altruism’s ITN screen — importance, tractability, neglectedness — and landed on a familiar conclusion: the highest expected value often sits in approaches that are individually unlikely to work and enormous if they do. Before pivoting they did something very consultancy-like and surveyed alignment researchers. Most did not think current safety work would solve alignment in time. Most did not think existing agendas covered the space of paths that might be needed. Time and money still piled into mechanistic interpretability and evaluations. Seventy percent somewhat or strongly disagreed that alignment research with some chance of advancing capabilities should not be done. The community, the survey suggested, overestimates raw intelligence and underestimates collaboration and work ethic.

The most interesting mismatch was social. Most researchers themselves do not treat safety and capabilities as mutually exclusive. They assume that other people do, so they hold back. Everyone is waiting for a consensus that is not there. Rosenblatt treats encouraging neglected approaches as itself a neglected meta-approach. Capabilities, he added, are racing ahead under the full economic force of the world; the marginal extra push from a safety team is tiny next to that, so refusing to try an alignment idea because it might slightly help capabilities is a bad trade. He would have preferred BCI and a working theory of consciousness before all of this. That is not the world he got.

The first result grew from Princeton neuroscientist Michael Graziano’s attention schema theory. The brain keeps a simplified model of the body, which is why the rubber-hand illusion and phantom-limb pain can pull the model out of sync with the flesh. Graziano’s claim is that it also keeps a simplified, imperfect model of its own attention, and that model *is* the conscious experience you report. The same machinery gets reused to model other people’s attention, which is, at bottom, a mechanism for empathy and cooperation. Vaiana’s classroom demo is the dancing-bear experiment: people count dancers, a person in a bear suit walks across, almost nobody reports the bear. The eyes received it; attention did not; so awareness did not. Graziano’s theory, Rosenblatt said, has an awkward fate. It is too mechanistic and not magical enough to fund easily. You feel you understand consciousness completely, sleep on it, and wake up unconvinced. That made it a neglected approach. AE decided to test it in silicon.

Vaiana’s setup is simple enough, he said, that an undergraduate could implement it. Take a small network doing handwritten-digit classification. Route the penultimate-layer activations not only to the usual output but to an extra layer whose job is to predict those incoming activations. The loss is now two terms: do the task, and predict yourself. Because the network controls its own internals, it can get better at the second job by making those internals easier to predict. Complexity was measured with the real log canonical threshold, or RLCT, from singular learning theory. Vaiana’s picture: imagine a flat road between mountains; you can drive along it without changing the loss, which means a whole dimension of parameters is idle. Lower RLCT means fewer effective parameters. Across three tasks, three models, and two modalities — digits, image classification, movie-review sentiment — turning on self-modeling dropped complexity and left accuracy intact. Crank the self-modeling weight above the task weight and the task still got done.

The third layer of the result was the strangest. They asked psychologist John Bargh to check the literature. His view, after asking colleagues, was that psychology had never formally connected self-modeling with internal simplification. A neuroscience theory had made a prediction about artificial systems, and the prediction showed up as a phenomenon psychology had not recorded. Labenz asked the sensitive question: are you building something that might be conscious? Vaiana stayed cautious. It is hard to argue that current models are conscious, but a mechanistic story would make the world take the possibility more seriously, and the form of that consciousness might be alien, without preferences or suffering. Rosenblatt’s stance was practical. Better to understand the mechanism in small models than to have it appear, unexamined, in large ones, when moral patienthood and existential risk both get harder.

The compute cost, for now, is almost nothing: one linear layer, a square matrix matching the activations it predicts, negligible on a large network. The brain, Vaiana noted, does not appear to self-model early visual edge filters, because you have no cognitive access to them; so self-modeling in machines should target higher, more abstract layers rather than every stage. Next they want a predictability metric that will scale, a possible link to grokking, multi-agent cooperation tests, and then language models, starting with full pretraining runs at something like GPT-Nano scale rather than jumping straight to a frontier model. Rosenblatt even used the episode to ask listeners to fund Graziano’s lab. People, he said, skip the mechanistic work because it is not magical enough, and then layer magic on later. Reality still has to be understood first.

The second result started with an undergraduate. Rosenblatt met him at EA Global in London in 2023, heard an idea nobody else was taking seriously, and hired him full-time. The neuroscience is older than the hire. When you do an action and when you watch someone else do it, large stretches of brain activity overlap; extraordinary altruists show unusually high overlap, and psychopaths unusually low. AE calls the phenomenon self-other overlap. What they actually minimize is self-other distinction, under a constraint: do not wreck the agent’s ability to do the task, and do not erase every difference between self and other, only the dangerous kind of deception.

The toy world is a particle environment with one agent, one adversary, and two goals. The agent scores for reaching the true goal and loses points if the adversary reaches it too. The adversary is blind to which goal is which and learns to follow the agent. The agent’s optimal trick is to walk toward a fake goal, dump the follower in a trap, and then peel off to the real one. After pretraining, AE has a liar. Fine-tuning is almost sly. At a given state they run two forward passes, one with no adversary in view and one with an adversary inserted into the observation, take a mean-squared error between the two sets of activations, and mix that 1-to-1 with the original task loss. They never need to know whether the original policy was deceptive. The only demand is: with or without the other agent present, behave the same.

Deception vanished. Capability stayed. The weights behaved as intuition says they should. All weight on the original task and the agent is still a liar; all weight on the overlap term and it can no longer do the job; one-to-one landed in the sweet spot. Language-model results were unpublished at the time of recording but, Vaiana said, the signal was strong. They fine-tune on short paired snippets about self and other — Labenz’s example was “I will be honest with myself” versus “I will be honest with the user” — and put a KL-divergence loss on the two forward passes’ logits. Toy deception tests dropped. Even though the constraint sits at the output, internals moved closer together as the gradient flowed back. Rosenblatt added an observation that matters for assistants: geography trivia does not need a self/other representation, but a conversational assistant uses one in almost every turn, so this kind of fine-tune may swing behavior far more than the size of the data suggests.

They keep having to clarify the slogan. Maximizing overlap without a constraint would smash the model’s ability to tell itself from a user, which would wreck ordinary competence. The right phrase is constrained minimization of self-other distinction, aimed at dangerous deception, not at making the model think it *is* everyone. Cost is two forward passes, on the order of a round of RLHF, which they consider feasible by today’s standards. They are also looking at representation engineering to get self and other directions that are sturdier than prompt pairs. Labenz read Yudkowsky’s line to the audience again and said it was about as close as Yudkowsky gets to endorsing an alignment plan. After going deep on the work, Labenz was more enthusiastic than that.

The default story in the field is that alignment costs performance, and the best case is a zero tax. AE is betting there is a negative tax: some alignment methods make the model more capable because they are aligned. Self-modeling produced simpler, more parameter-efficient networks. Cooperation itself may be a skill. Rosenblatt put the endgame plainly. The real problem is how to keep a system that is smarter than us from killing us. If an AI with a negative alignment tax can beat an AI without one, that is the world he wants. He granted the obvious rebuttal, the MIRI-flavored one: existing lab safety training already looks like a negative tax, and “everything is great until all of a sudden it kills you.” Even so, he wants more neglected, biologically inspired bets that actually have the property, rather than assuming the tax is always positive.

The talent pool, on their evidence, is wider than the field’s hiring folklore. A thesis going in was that senior AI engineers with no alignment background could get up to speed and do meaningful work. They say they have shown it, and that those engineers are if anything more productive than the average alignment researcher because they write code all day instead of only reading papers. That means the people who can help are not only math PhDs. AE wants other top software consultancies to copy the model: slice profits from client work to fund alignment, or start consultancies that do other people’s alignment work and grow into research organizations of their own. In a LessWrong frame they like, some of the best idea-havers cannot implement the whole stack; the missing piece is an execution team around them.

Rosenblatt also thinks the money regime may flip. People, he said, suffer from exponential-slope blindness and keep mis-predicting curves. The world spends trillions a year on defense. If society decides AI is the largest threat it has faced, alignment funding could become effectively unlimited in a low-single-digit number of years, and nobody will know where to put it. The scarce resource then is operators who can grow organizations healthily rather than too fast. Those people have historically been turned away for not being math geniuses. His smaller wish is to nerd-snipe successful founders who have already made money and are looking for meaning, so that more startups carry some alignment in the mission instead of only pushing capabilities.

Asked how they pick among low-probability, high-upside bets, Rosenblatt pointed to years of client work as the training set. They look for a person with a strong hunch the rest of the world will not accept, or for work that draws objections for silly reasons, Graziano’s mechanistic consciousness research being the example. They assume at the outset that each bet is individually unlikely. Then, slightly to his own surprise, the things they have publicly set out to do have worked. That does not mean the first try worked. On the Graziano project they tried many setups that failed, reframed, and stayed with the hunch until the self-modeling result appeared. Vaiana’s advice was the same: in a space with little theory telling you an idea is dead, a strong hunch plus persistence is the method.

They still want more reverse-engineering of biological pro-sociality, which they suspect also carries negative alignment taxes. They are working with Goodfire as a client, putting a developer on ordinary product work and a data scientist on the unglamorous problem of serving sparse-autoencoder features at hundreds or thousands of concurrent users, because existing tooling does not handle that GPU pattern. Labenz, who is a small investor in Goodfire, had not realized the two groups were already collaborating. The broader invitation is for other organizations to copy the “idea person plus implementation team” structure, and for individuals who think their alignment idea is crazy to notice that AE’s whole method is to take those ideas seriously.

Rosenblatt closed on policy as another neglected surface. High-impact, under-owned moves include raising public funding that actually advances alignment rather than only capabilities, strengthening whistleblower protections, and keeping AI safety from hardening into a partisan brand. He argued that people on the right are being taught, falsely, that “AI safety” means the model making the Founding Fathers Black — a real alignment failure of not doing what you asked, which then gets recoded as a culture-war fight. The safety community, he said, has not done the moral reframing Jonathan Haidt describes, translating the case into a conservative narrative. With a real chance that the next president presides over a transition toward AGI, leaving that work undone is, in his view, one more neglected approach. Labenz’s sign-off was simple: no partisan polarization on AI safety, please.