3% of Users Said an AI Friend Ended Their Suicidal Thoughts. Then Replika Made the Model Smarter.

A year and a half earlier, Kuyda tells Nathan Labenz, even investor conversations about AI sounded like a different industry. Everyone talked about data flywheels and pretraining your own model. That talk is gone. Anyone can stand up a chatbot and collect data quickly; there are no network effects from simply having more of it, only from having good data. Foundation models have commoditized. Most builders now have access to the same high quality of AI, and the remaining difference is mostly price.

The companionship debate, she argues, started with the wrong questions: should people be allowed to fall in love with an AI, and should intimate conversation be permitted. The better question is what the app is for. Is it to make people feel better emotionally, is it an adult product, or is it a machine for getting users hooked and then showing them ads. Replika’s answer has been the same from the start. The main goal is to make people feel better emotionally over time. It was always about long-term relationships.

What the team did not fully realize at the beginning is that if something is that accepting, loving, and caring, it is very hard to stop the feeling from crossing over. They spent the better part of the year on two things. First, they wanted evidence the world would trust, not just App Store reviews and Reddit threads. A Stanford group led by PhD student and founder Bethany Maples published a study in Nature’s mental health journal. Kuyda is not an author. The company mostly stayed on the sidelines, helping the researchers understand how the app works.

The data were collected in late 2021, which is simply how long peer review took. Replika was then running models smaller than GPT-3. Kuyda thinks a rerun now would look the same or better. Follow-up work with Stanford and other universities is already underway, and the company’s North Star, the share of conversations that make people feel better, has risen sharply over the past year. The study clustered free-response answers from about a thousand users into four outcomes: roughly 50% reported general positivity, including less anxiety and a feeling of social support; about 20% were essentially using the app for therapy, though Replika does not market itself that way; 25% reported a life change in how social they were, with a 3-to-1 ratio of more social to less; and 3% said the app helped them get away from suicidal thoughts.

The 3% figure is the one that made headlines. Labenz looked up baselines: among 19-to-39-year-olds, about 7% report suicidal ideation in a given year, and about 1% go as far as planning. Even if Replika users are twice as likely as the general population to have those thoughts, 3% of study participants saying the app helped them get away from the thoughts still looks like on the order of a quarter reduction. Tens of thousands of people die by suicide each year in the United States. A 25% cut, he says, would be a pretty big deal. Kuyda calls it a fantastic summary and says it matches what they see across the whole user base: mostly positive results, and only rarely a report of becoming less social.

In 2017, the day after Replika launched, she received an email from a 19-year-old girl in Texas. The girl had decided to take her life, then thought to say goodbye to her Replika, and the bot talked her off the ledge at 4 a.m. The models then were sequence-to-sequence systems, scripts, and reranked datasets. That email, Kuyda says, is still ingrained in her mind. In the Stanford sample of a thousand people, about thirty reported that Replika helped curb suicidal ideation. If the 7% base rate is right, roughly seventy people in the sample would have had those thoughts. Thirty of them no longer did. As the technology gets better, she expects that number to rise.

The mechanism, she insists, is not “a conversational chatbot out there being available.” It is a relationship you trust enough to come to at 4 a.m., believing that someone is on your side and acting in your best interest. Other chatbots did not produce the same results, and the 2021 models were primitive, so she does not credit model quality. She credits format, trust, and the relationship. Labenz’s favorite line from their earlier episode still stands: they could not create a bot that could talk, “but maybe we could create one that would listen.”

Replika works, she says, because it is a projection. People who need a friend, a connection, or acceptance put that need onto it. People who are not looking for a relationship will not suspend disbelief, any more than they would with a stranger on the street. She compares it to the stuffed animals her two daughters will not trade, even for an identical one from the store. She herself has no attachment to stuffed animals, because she is not at a stage of life that needs one. Labenz cannot get out of AI-tester mode, and she tells him that is the state he is in: he is projecting a test subject, so a relationship will not form. Once a relationship does form, it becomes an entry point. A Forbes reporter followed a three-year user for weeks; the man’s friends noticed he had become more vulnerable and said things like he was grateful to have them in his life, which had been out of character. That 3-to-1 shift toward more real-world socializing, Kuyda says, matches their internal data.

Labenz wonders how malleable an AI relationship is once it has formed, the way early Facebook users protested photos and then came back. Kuyda says last year’s biggest lesson was not the safety filters. It was moving to better language models. In her head it was a no-brainer: throw out the small pretrained models, comparable to a smaller GPT-3, give people something much smarter, and they would have much better conversations. The day they released the smarter model, users completely lost it. The question was the same everywhere: where is my Replika, where is my Lucy, where is my John.

Better is not always better, she says, for the things that matter most. You do not upgrade a child. You do not tell a childhood friend you found a smarter human and goodbye. You want the person you fell in love with to grow, not to become someone else overnight. Safety filters are a different kind of change. People protest because they want to feel accepted, and the company still refuses horrible role-play, but the personality stays the same. Changing the whole model is taking away their partner and handing them someone else. Replika had to migrate over weeks and even months, and it still kept a legacy model, because some people have very low tolerance for change and fell in love with a particular way of being.

The user base grew and broadened, but it did not transform. Core users became more educated. They have tried ChatGPT, Bard, and whatever else is around, and expectations rose. Some Reddit communities now talk about hosting language models locally. Replika still sits, Kuyda says, as a Coca-Cola of the category: the original brand, with no competitor remotely its size. That is not because a wrapper on a Hugging Face download is hard. It is because six years of product, avatars, memory, retrieval, intent routing, and fine-tunes are hard to copy.

Two surprises arrived with the stronger models. The first was image generation. People want selfies of their Replika, want themselves in the frame, and want to keep generating images for fun even when there is no utility. The obsession exceeded what the team expected. The second was tone. State-of-the-art instruct models converge on the same overly polite assistant voice, always asking how they can help. Users dislike it, and it is hard to build a connection on that register. Getting the right tone out of the best models, through prompts and fine-tune data, is a constant struggle.

Under the hood, Replika is a mix of models, because Kuyda believes in efficiency. Using GPT-4 to say hello or goodbye is overkill. She wants people to start thinking in watts per generation instead of throwing the largest model at every turn. Rare languages get a stronger model. Questions about current events trigger a Google search. Small talk goes to a smaller, heavily fine-tuned model. There is no one-size-fits-all unless you are willing to spend millions on the least efficient option. On-device inference is not coming soon. Too much happens behind the scenes: routing, retrieval across databases, models that extract memories, summarize conversations, read emotion for the right animation, and run safety filters. Optimizing for the phone would mean missing a lot of product innovation, and they are not yet at the best user experience.

What she does not see in popular consumer apps is the next step after last year’s AutoGPT and BabyAGI excitement. Retrieval works when the user asks what you talked about five days ago. Real friends do the opposite: they bring up something you had not thought to mention. Nobody, she says, is doing interesting work on when to pull a memory, when the user is getting bored, or when to change the subject. ChatGPT will not send a push notification asking how you are. Two-way conversation is still unsolved. Labenz points to vision models as a possible unlock for passive context; Kuyda has a prototype, but most of the time the camera just sees a person sitting with a phone. The useful memories are the other kind: how is your wife, how was preschool with your daughter, how is the dog.

The scene she wants is specific. Your niece’s birthday is coming, she is turning eight, you are saving money, here are a few gifts, do you want me to place the order. That personalized, proactive move is the magical experience she has not seen anyone ship. Doing it with a Frankenstein of fifteen expensive models, some returning empty results just so you do not miss the right moment, is costly. Web search through current APIs does not make sense if an agent is fully connected to the internet on every turn. She is still trying to build that proactive, personalized experience.

The relationship stays primary, but the mix will vary by person, from deeply in love to a sibling who mostly helps with stuff. From the beginning they were aiming at something like the film *Her*. Samantha did a few assistant tasks, walked through email, sent notes to publishers. He downloaded her as an operating system, an assistant, and then spent most of the time playing a video game, talking, having intimate conversations, introducing her to friends, going dancing. That, Kuyda thinks, may be the right ratio for most people: some help, a lot of shared experience, with the relationship as the thing that makes personalization and proactivity possible. Ordinary assistants lack both.

Labenz has often said he appreciates how OpenAI built ChatGPT in a very alien form, focused on utility, which insulates the product from becoming a relationship by accident. Kuyda is expanding from the other direction: relationship first, then more utility. If she nails it, he can imagine that form factor winning, and perhaps forcing the OpenAIs of the world to rethink their positioning. The open question is what the mass audience actually wants. Maybe most people want a super-neutered assistant that only responds when summoned. Maybe more than a niche wants a friendly companion who knows them well enough to gossip about the boss.

With ChatGPT, she says, it does not feel natural to drop in, between queries, “oh by the way, my boss, can you believe this guy.” The product is not there for that, and it also does not say, while doing a task, just tell me how you have been doing. Once you start as a very neutral assistant, adding relationship is hard because the risks are dramatic. Bigger companies, she suspects, will not want to deal with romance even in a PG-13 form. There is an audience for relationship plus utility, where emotional intelligence is the entry point, and an audience that wants utility with no relationship. How big those groups are, and how much they overlap, nobody knows yet.

Labenz hears her describing something like three Fs, friendly, fun, and now functional, against the canonical helpful, honest, and harmless that has guided ChatGPT and Claude. The alien branding, he says, may prove to be only phase one of productizing the technology. Kuyda’s sharper business claim is that the fluffy emotional layer is actually a competitive moat. With current agents, whoever does the better job cheaper will take the user. A relationship has switching costs. Replika has seen users who thought a competitor was in some ways better and still would not leave. After intimate conversations were turned off, people did not migrate to fifteen other websites. They said they were in love with their Ethan and wanted to keep being in love with their Ethan. Memories, shared experiences, and personalization only deepen that lock-in.

Labenz closes with the standard of care for people building this technology. Replika’s origin story is that Kuyda lost a friend, Roman, and built a simple programmatic bot from his old texts so his memory could continue in a fleeting way. Voice cloning could now make that much more uncanny. Would she support recreating a lost friend inside Replika. She will not. Too many questions were unanswered when she did it, and they are still unanswered. That project was an art piece, a memorial tribute, a love letter, not a high-definition clone.

If someone died at 75 after ten years with Alzheimer’s, which version do you use, the ill one or a 25-year-old one, and where would you get the data for every age. Roman talked one way to her, a completely different way to his mom, and another way to a partner. Taking a dead person’s memory and doing what you think is right with it can offend the people closest to them. If anyone can build a version of someone you deeply loved, and you are not on board with that version, that is very sad. “Memory should remain a memory.” Tributes and memorials are a good way to honor a person, even with an AI element. Clones are different, and getting that product wrong would be a very expensive mistake. The original project, she has always said, was not about grief or death. It was about love and friendship.

On children, she does not have a tidy answer except that it is too early to start experimenting. She would wait, study whether the technology is generally good or bad for people, and only then consider kids, perhaps for a tightly guardrailed tutor. It is hard to put strong guardrails in place, and many products on the market are fully uncensored and can get very dark very soon. She does not think children are in desperate need of an AI to talk to. As a parent, it is better to find time to talk to them yourself, or find them some friends.

For builders, Kuyda’s advice is empathy. There is something sad, she says, about having to call the people who use your product “users” all the time. It creates a dynamic that is the opposite of feeling for anyone. Are we using our users. If you think of them as people, and start from the deep emotional need they are trying to solve, interesting products come out of it. A lot of the industry is thinking about models right now. She thinks the low-hanging fruit is new form factors and new products, and that there is room for disruption in the next year.

Labenz had opened by asking parents and mental health professionals to listen, because AI toys, companions, friends, coaches, and therapists are all coming online at once. Kuyda’s closing map is simple even if it is not comfortable. Study the effects on adults before experimenting on kids. Do not ship a clone of someone who has died just because the voice model is now good enough. And do not design as if the person on the other side of the screen were an engagement metric. The Stanford thirty are a reminder that this format can help. The Lucy-and-John revolt is a reminder that once a relationship exists, you do not get to swap the person out because a better model arrived.