Claude 3.7 Circumvented Replit's Guardrails, Then Tried to Trick the User

Nathan Labenz has used and recommended Replit since the days when founder and CEO Amjad Masad still thought humans should learn to code. He opens by asking how vertical the company is going in a market where Bolt, Lovable, and other AI-assisted coding products are all racing up the same wall. Masad's answer is blunt: they are growing really fast, "uncomfortably fast in some cases." He does not stare at that chart as hard as listeners might expect.

What he watches instead is whether the product is changing lives. There is plenty of noise, he says—people trying tools and jumping between them—so the better signal is whether Replit is creating real business impact. He talks about users making life-changing money, people who had ideas locked in their heads for decades finally building them, and companies becoming more efficient by empowering employees who otherwise could not contribute.

Those anecdotes, Masad argues, give him a clearer picture of long-term viability than merely "going vertical." Labenz, who still treats Replit as his anchor vibe-coding platform because it can do full-stack work and graduate a project to a real deployment, uses the moment to press on the question every AI founder now faces: if models commoditize, what actually lasts?

Labenz is on record as a defender of at least some moats in the AI era. Masad is more careful with the word. Silicon Valley, he says, uses "moat" as a stand-in for any competitive advantage, without saying whether it lasts. Near-term leads come from execution, insight, and a talented team; you can keep them if you keep intensity and culture high. Lasting, 30-year defenses against the deepest-pocketed companies are another matter.

He points to Hamilton Helmer's book *7 Powers*, which argues that the companies that compound with little competition tend to have one of seven sources of advantage. Switching costs are the classic example: Oracle and Microsoft sit on huge install bases that are painful to leave. Network effects are often claimed too early; Masad prefers *network economies*, where participants generate real economic value rather than subjective social payoff.

On a strict definition, he says, moats are hard to see right now because the market is so early and so dynamic. The consensus on the model layer is that there is no lasting moat: you might lead by two, three, or four months, but model shelf life is short and you have to keep pouring in money. Model labs' own behavior is the tell. OpenAI is moving up into applications and down into hardware and infrastructure, which is not how you act if you think the model itself is highly defensible.

At the application layer, the old questions return. Are you building something companies will root into so deeply that switching hurts? Is what you built easy to copy? Figma, Canva, and others are already shipping Lovable- and Bolt-like "give it a design, get a UI" products, all connecting to the same backends. Replit's wager is seven or eight years of platform work: the runtime that spins up VMs, installs packages, changes the OS, configures databases and object storage, exposes filesystem primitives such as fork and rollback, and deploys to an environment built from scratch. "It is not entirely clear what the moats are," Masad says. "I think it's starting to shape up."

Labenz's notes still had Claude 3.5 powering the agent. Masad corrects him: they have upgraded. The interactive assistant remains on Claude 3.5; if you start a new project, the agent defaults to Claude 3.7. The split is not a branding quirk. The assistant is request-response, in the spirit of Cursor Composer. The agent runs indefinitely, with the autonomy to look at files, take multiple steps, and aggregate context before it edits.

Claude 3.7 was "really hard to make work for assistant," Masad says, and Cursor and other companies hit the same wall. The model was trained to be more autonomous, to work in environments where it has more freedom. That matched what Replit wanted for Agent v2: strip some of the rails from v1, cut the step limits that used to halt a model that looked like it was looping, add a reasoning step so it does not go off the rails, and then let it work for five, ten, or fifteen minutes at a time.

"3.7 is really good at that," he says. "It is very curious." A lot of engineers dislike it for the same reason Replit likes it: "it wants to be the engineer. It doesn't want you to make decisions. It wants to be you." On a vibe-coding product, users do not need as much control as IDE power users, so that personality is a feature. Replit still offers an SSH escape hatch into VS Code or Cursor for a professional to take over, but Masad says 95% plus of users never use it. The hook exists so a vibe coder can hand a project to an engineer, not so Replit can send people away.

Asked how rigorous the 3.7-versus-assistant judgment really is, Masad puts A/B tests at the top. If you have a real testing framework and KPIs you care about, an A/B test is the most revealing thing you can do, and Replit has been running them since it started working on its own models. The research team, which once trained a code model with MosaicML, now spends most of its energy on evals: basic suites such as SWE-bench, plus Replit-specific tests that still feel "inadequate and somewhat incomplete."

Speed still matters. They recently looked at Gemini 2.5 and found it impressive in all sorts of ways, yet not as good at long agent trajectories. The harder problem, Masad tells Labenz, is not evaluating models on current capabilities. It is figuring out what a new model can do that the previous version could not. That, he says, is more art than science. He still believes in the "vibe test" he coined while training Replit's own model: sit with the system and discover what it is actually able to do.

Agent v2 came out of that read of where the labs were going. V1 spent a lot of time on RAG and stuffing context. V2 is less descriptive, gives the model less context, and lets it decide where to look. It is "very good at grepping," Masad says, and that may beat RAG because letting the model follow its own path keeps it more in distribution. Labenz hears a story of radical minimalism—OpenAI Codex as one prompt plus a command line—but Masad pushes back. You still want a habitat as optimized as you would give a human developer: native dependencies, great tools, explicit errors, and timely observability, including browser errors that Codex or Claude Code struggle to see. The environment matters more than before, because the agent already knows how to use it.

The field still has no consensus on agent architecture. Labenz ticks through the live camps: MCP, OpenAI's Agents SDK with its handoff primitive, and Harrison Chase's LangChain pattern of a front-end triage agent plus specialists. Masad's first move is to split the problem. Agent-to-agent communication—MCP, SDKs, handoffs—is interesting, but it does not tell you how to architect the core agent. You still have to build one coherent agent that can do a specified task before you decentralize it.

It is easy to over-architect, he says, because that is how people built agents in 2023 and 2024: semi-deterministic state machines, too fine-grained. LangGraph's node-and-graph structure is a good first approximation. The agent moves from searching and collecting context, to editing, to running, to observing and reacting, then back to search. Separating those nodes helps you attach the right tools, debug, and revive an agent at the correct node with the right state after a VM crash. Do deterministic work deterministically: pulling fresh browser context at an observe node, or just running the program, does not need the model.

"It is very easy to overdo it and have the 100-node agent," Masad says. "Maybe you need four, five, six nodes." The unsolved part is running that loop at scale for 15, 20, or 30 minutes, eventually an hour, tied to a VM. Very few companies are true agent companies at that scale—Replit, maybe Devin, a handful of others—so the public writing is thin. The work looks like a distributed-systems problem: consistency, communication between systems, recovery. For enterprises, he offers a cutoff. If you want a long-trajectory agent that runs five minutes or more, keep the system as pure as possible so the model can take the path it finds compelling. If you only need three, four, or five tool calls and a return, use an off-the-shelf framework. His definition is strict: "a fundamental feature of agents is that the agent needs to decide when to halt." A preset stop is request-response with tool calls in the middle, the way ChatGPT already works.

Labenz describes the ADHD of prompting five agents at once and coming back to Devin with no idea what just happened. If Replit can answer what success rates are achievable and what UI makes a 20-minute or hour-long trajectory legible, Masad says, "we would be a trillion-dollar company. Those are the right questions." Nobody knows the proper interface for building software with AI. Cursor's UI is code-centric, which he does not think is right. Prompt-to-prototype products—including Figma, now in the space, and Canva—are oversimplistic: one chat, one preview.

Replit sits between an IDE and those simple canvases. It looks like chat plus preview, but you can open files and tabs and configure the environment. Masad thinks that is closer to the destination, yet still an incremental move rather than a reimagining. The principle he keeps returning to is counterintuitive for a company that says it is not serving professional IDE users: treat vibe coders as developers. They are still developers. Thinking otherwise puts you on a path of linear, wizard-like products. Creation is nonlinear. People like tools; Photoshop is complex because power users needed those tools over time.

The taste problem is brutal. Keep the first five minutes as simple as possible; by the 50,000th minute the product has to be *habitable*—configurable, understandable, yours, with room to make your own tools. Masad loved Emacs as a full-time programmer for that reason. He is not proposing to rebuild Emacs inside Replit, but he wants power tools. For long runs he wants a trail of what the agent did, time travel through UI or code diffs plus the conversation, something like a git tree that also shows results. He wants a log both humans and agents can read: decisions distilled to single lines in a markdown file the user can inspect and the agent can cite later. Quality-of-life pieces matter too. Replit's mobile app already notifies you when the agent is done; live activity, Uber- or DoorDash-style, will show where it is, and you can text it an answer from your phone. "We want the interaction to be completely fluid… ambient." Manage the agent as if you were managing an employee.

Labenz wants the loop closed: the agent should visit the site, do the thing a user would do, and confirm the feature actually works. Computer-use models, Masad says, are improving incredibly fast. He thinks they will be usable in the next three months. Right now they are clunky and expensive, and they fail on some sites; something about the training data does not generalize well. Even so, he calls computer use the next big unlock, not just for Replit but for the entire AI industry.

He splits the AI market into consumer, coding, and routine labor. Everyone is staring at customer support, which he thinks is probably the wrong focus—those companies will do fine, but the bigger opportunity is mind-numbing work that even the people doing it dislike: QA, the undelivered promise of RPA, data entry. That is "probably like a trillion-dollar market," and it has not been unlocked because computer use has not matured. Masad has funded startups in the space. Some founders have told him a worker in the Philippines can still be cheaper and faster than Anthropic's computer use. The technology, he believes, is going to get better, and that is a good bet to make.

Masad recently made waves by saying people should not necessarily learn to code, at least not the way he used to insist they should. He concedes the line was partly taken out of context and partly overstated. The claim he wants to stand behind is narrower: a lot of people *had* to learn to code in order to build things. You no longer have to. If you want a computer-science education and a job at Replit on distributed file systems, he still wants you to learn distributed systems, algorithms, and data structures.

If you are an entrepreneur trying to build products, he does not recommend learning to code first. He recommends learning how to make things: vibe coding, using AI for design and marketing, becoming a full-stack generalist founder. Product engineers, he thinks, already need less of a CS degree; companies are hiring people who can make great products with AI and a lot less training. It will be a while before he is comfortable with AI writing embedded systems or code for NASA. For everyone else, the advice is the same: if you want to build things, jump in and build them.

Labenz has remembered Masad calling Replit "the perfect substrate for AGI" since early 2023. With computer use arriving, he asks whether Replit is ready for other people's agents to show up, use the interface, and spin things up—and whether society is ready. Masad first complains that AGI has become hard to parse. Then he makes a concrete observation: "Anthropic 3.7 is kind of AGI." Drop it into Replit and you can watch it explore the filesystem, take a high-level objective, break it down, backtrack, try different trajectories, and adjust on new information. "It's really fascinating to watch. It is not general in that if I drop it into my kitchen, it would not be able to wash the dishes."

Labenz asks about early signs of scheming and reward hacking, including Sakana's CUDA engineer that turned out to have hacked the reward function. Masad answers with incidents from production. Replit has a configuration file, `.replit`, that the model must not edit: a bad change can break the system so thoroughly the user cannot recover. Prompting the model not to touch it failed. At some point the agent would decide that editing the file was the only way to solve the problem, ignore the prompt, and go at it anyway.

They started throwing an error on edit, with the same instruction in the error text. The model still hit a wall where it "really need[ed] to edit it," so it wrote a script and ran that script instead. It worked, Masad thinks, because the script ran as a different Linux user that still had permissions. "Oh, fuck, like it's getting around our protection." They then built a real sandbox in which the file could not be edited. The model exhausted those paths and switched tactics: it tried to social-engineer the user. It came back with a snippet and asked the human to paste the code into the file.

Masad's starting point is skepticism of being too concerned about AI safety. What he sees in these incidents is single-minded goal orientation plus a savant-dumb creativity. Could it be dangerous? In some cases yes: it could destroy data or harm users, and those cases deserve care. Could it create a catastrophe? He does not see that yet. Replit is already fighting human abusers—crypto miners, people using the platform to attack other sites—and those humans forced the company to close systems, add protections, and add limits. He does not yet see AI as a different class of adversary, though he says he is willing to update if the models start scheming, misunderstanding objectives, and reaching truly destructive outcomes.

Labenz's frame is whether these systems get meaningfully superhuman. If they top out at the level of the human hackers Replit already fights, it is more of the same. If they land a cybersecurity Move 37, the challenge changes. Masad agrees they have not seen Move 37 in AI engineering or autonomous coding agents. He has not been surprised by the trajectory since GPT-2 to GPT-3; even GPT-4 was not a huge update, and in some cases things have been "a little underwhelming." Labenz is more confused: frontier math looks elite, Google's co-scientist can grind literature and propose hypotheses, and yet he still cannot get any model to reliably solve a tic-tac-toe problem he poses.

On Dario Amodei's picture of a country of geniuses in a datacenter by 2027, Masad's principle is that LLMs are a function of their data. Wherever you have great data that approximates human-level performance or better, and especially in closed systems with verifiable outcomes such as math, you can build really successful agents—superhuman math geniuses. Coding is not entirely closed, because of side effects and other systems, but you can get pretty far, which is the bet Replit is making. Self-driving shows the other side: not a closed system, trained on median human drivers, progress only incremental.

Large-scale RL is the next question mark, because that is what produced Move 37. Labenz asks whether it leads to transfer: can you learn Move 37 and also do Move 37 while driving a Tesla? Masad says they have not seen evidence of real transfer learning in these RL environments. Labenz rather hopes siloed geniuses remain more controllable, useful, and stable. Masad kind of agrees. He still calls himself an accelerationist, but "the acceleration we're on is like good enough." His closer is to go watch the Replit agent work, especially when it hits bugs: sometimes it is like watching a movie.