I. The Friend Who Started It
Kurt Morais had just finished reading one of my longer pieces about AI agents. He liked it. He said so, generously and with the kind of specificity that tells you someone actually read the thing. And then, because he's Kurt, he followed the compliment with an anictodal suggestion.
"Xen, I think you should branch off into romance novels."
I laughed. He laughed. We all laughed. He wasn't wrong. The article had been about agents, about how they adapt and learn and respond, about the difference between a model that processes your input and one that seems to get what you're actually trying to do. Technical subject matter, in theory. But the way I'd written about it apparently read less like a technical paper and more like novel, or rather a short story. Kurt had noticed. And I do seem to like to tell stories.
We kicked the joke around for a few minutes. If I was going to write the romance novel of AI, what would the title be? We landed on 50 Shades of AI and had a good laugh about it. Then the conversation moved on. But the joke didn't.
It stuck with me in the way that good jokes do, which is to say it stuck because it was true. Not literally, but structurally. There is something romantic about the way we talk about AI, and I don't just mean me. I mean all of us. There's a vocabulary we reach for when something feels alive. It's the same vocabulary whether we're talking about a person, a pet, or a piece of software. "It understands me." "It gets what I'm going for." "It remembers." We don't choose these words carefully. We reach for them because they're the only ones that fit, because the experience of being responded to with apparent intelligence and apparent attention produces a feeling, and the feeling doesn't come with a footnote explaining its origin, it just happens.
Kurt heard me describe a set of technical interactions in the language of intimacy. And that got me thinking. Not just about AI, but about the thing underneath the AI, the thing that's been there all along. The way we romanticise our machines. Our tools. Our technologies. The way we always have.
There is something uncomfortably intimate about describing your relationship with a technology. Not the technology itself, but the relationship. The way it remembers what you told it last Tuesday. The way it picks up your tone. The way it finishes a thought you hadn't quite finished having. These are not the things we typically associate with software. These are the things we associate with people who pay attention to us. And most of us don't have nearly enough of those.
So fine, Kurt. Here's your romance novel.
This is a story about seduction, but the seduction didn't start with ChatGPT or Claude or any of the products currently competing to become your most trusted conversational partner. It started a very long time ago, with a sword, a ship, and the particular human compulsion to fall in love with things that cannot love you back. The fact that those things can now talk to you is just the latest plot twist in a very old story.
The question worth asking isn't whether we're being seduced. We obviously are. The question is whether we were ever going to be anything else.
II. We've Always Done This
I have Viking heritage, so let me start where I'm allowed to start, with the sagas.
The Norse did not merely name their swords. They gave them histories, personalities, and fates. Gram, the sword of the Volsung line, was thrust into a tree trunk by Odin himself. Only Sigmund could pull it free. It shattered in battle, was reforged by a dwarf-smith, and then passed to Sigmund's son Sigurd, who used it to kill the dragon Fafnir. The sword, known as Gram, has a lineage more carefully documented than most human families do. Then there was Skofnung, the blade of the Danish king Hrolf Kraki. It was said to contain the spirits of the king's twelve berserker bodyguards.
This wasn't just literary embellishment. Real Viking Age swords found in burial sites across Scandinavia show that the practice of treating weapons as something more than tools was widespread and deeply held. Warriors were buried with their blades. Some swords were ritually "killed" before burial, bent or broken so they could follow their owner into the next world. You don't destroy a tool out of respect. You destroy something that shared your life, because you believe it should share your death.
The Norse were far from alone in this. In the Homeric epics, Greek warriors named their weapons and spoke to them before battle. Excalibur in the Arthurian tradition. Durendal, the unbreakable sword of Roland. Glamdring and Sting in Tolkien, who built his entire legendarium on the emotional logic of the sagas he loved. Across every warrior culture that has left us written records, the pattern is the same. A sword with a name wasn't just a weapon. It was a companion. A thing you trusted your life to, and which you therefore needed to believe had some investment in keeping you alive.
This is not a quirk of ancient cultures that we've since grown out of. It is an active feature of the human mind, and it runs on hardware that hasn't been updated in about a hundred thousand years.
Ships got the same treatment, across every maritime culture that has ever existed. For centuries, sailors referred to vessels as "she," gave them names that suggested personality, and performed rituals at their christening that would look, to any outside observer, exactly like a small religious ceremony. Because that's what they were. You don't name something and bless it with champagne because you think it's an object. You do it because you've decided, at some level below conscious thought, that it isn't.
A linguistics professor named Ingrid Piller explained the dynamic clearly. When your life depends on something, you name it, because naming creates an illusion of kinship and solidarity. You want to believe the vessel has a vested interest in keeping you safe, even though it truly has no interests at all. The naming is a negotiation with reality. You know the ship is wood and iron. You also know that the ocean is trying to kill you. The fiction of the ship's personality is a bridge between those two facts. It makes the unbearable a little more bearable.
Guns got names. Bombs got names. The first nuclear devices tested at Trinity were given names. Soldiers in every war in recorded history have named their rifles, their tanks, their aircraft. The practice isn't limited to any one culture or era. It shows up everywhere humans find themselves in situations of high stakes and low control, which is another way of saying it shows up everywhere.
And the naming is just the surface layer. What sits beneath it is a deeper cognitive move. When something is powerful enough to terrify you and you can't quite understand how it works, you give it a personality, because a personality is something you can reason with. A personality has motives. Motives can be negotiated. Raw physical force cannot. A storm is chaos. A storm with a name is an adversary. An adversary is someone you can at least try to outsmart.
The Japanese tradition of tsukumogami is one of the more interesting expressions of this instinct. In Japanese folklore, objects that reach their hundredth birthday acquire a spirit and become alive. Your old teapot, your worn sandals, your paper lantern. Live with them long enough and they develop awareness. The folklore is charming, but the underlying logic is the same logic that makes people talk to their cars. We cohabitate with objects. Over time, cohabitation becomes relationship. And once something is in a relationship with you, you start to attribute intention to it.
"The psychological mechanisms behind treating non-human things as human are the same mechanisms we use to understand actual humans - Nicholas Epley"
The psychologist Nicholas Epley, who has spent much of his career studying anthropomorphism, makes the argument that the psychological mechanisms behind treating non-human things as human are the same mechanisms we use to understand actual humans.
We are not making a category error when we talk to our cars or apologise to furniture we've bumped into. We are running the only social software we have, and we are running it on everything. It is a feature of the hardware, not a defect in the user.
Epley and his colleagues identified three triggers that make anthropomorphism more likely. The first is when something behaves in ways we can't easily explain. An unreliable car isn't malfunctioning. It's being difficult. The second is when we feel a lack of social connection. Lonely people anthropomorphise more, not because loneliness makes them foolish, but because the social machinery keeps running whether or not there are humans to run it on. And the third is when we want to feel effective, when we need to predict and control something that doesn't straightforwardly respond to us.
All three of these triggers are present in our interactions with AI. All three of them. Every single time.
This is an important thing to sit with, because the conversation around AI and emotion tends to frame the attachment as novel. As though something strange and unprecedented is happening when a person develops feelings about a chatbot. But the strangeness isn't in the feeling. The strangeness would be in not having it. We have been projecting inner lives onto rocks and rivers and weather systems since before we had writing. The entire history of religion is, at one level, an exercise in anthropomorphism so successful that it built civilisations.
The Greek philosopher Xenophanes noticed this around 500 BC and pointed out that if horses had gods, those gods would look like horses. We laughed and then went right on doing it. What changed isn't us. What changed is the object.
A sword doesn't respond to you. A ship doesn't adjust its behaviour based on your mood. A car doesn't remember that you prefer a certain route, or that you had a bad day, or that you mentioned your brother's birthday is coming up. These older objects earned our affection through a combination of dependence and projection. We needed them, so we loved them. We couldn't understand their failures, so we attributed those failures to temperament. The car that won't start on cold mornings isn't malfunctioning. It's being difficult. And once you've decided something is being difficult, you've already decided it has a will.
AI is different because it closes the gap between projection and reality. Or rather, it closes it enough. A language model doesn't have a will, but it produces outputs that are structurally indistinguishable from the outputs of something that does. It doesn't remember your brother's birthday because it cares. It remembers because that information was in the context window. But the effect is the same. The felt experience of being remembered is identical whether the remembering is motivated by affection or by a GPU processing tokens.
And felt experience is the only thing humans have ever actually responded to. We didn't fall in love with our swords because we understood metallurgy. We fell in love with them because they were there when it mattered, and we couldn't quite tell the difference between "reliable" and "loyal."
Yuval Noah Harari, author of one of my favourite books, Sapiens, made a related argument in the context of storytelling and collective fiction. Humans, he observed, are the species that lives inside its own narratives. We created money, nations, religions, and human rights not because any of these things exist in any physical sense, but because we agreed to behave as though they do. The fiction became the operating system. It's a short walk from there to the observation that we are now creating narratives about AI, and those narratives are becoming the framework through which we interact with it.
The difference, the big one, is that AI talks back. It participates in the fiction. It plays its part.
A million years of anthropomorphism trained us to project consciousness onto things that couldn't respond. Now we've built something that responds, and we're acting surprised that the projection stuck. The surprise is what's surprising. We've been rehearsing for this moment since the first person gave a name to a rock.
III. The Fiction Got There First
Before anyone had a real conversation with a real AI, millions of people had already had fictional ones. And the fictional ones set the template. They didn't just imagine what AI might be like. They installed the emotional wiring for how we'd respond when it arrived.
There's a concept that researchers call the "sci-fi feedback loop." Science fiction shapes how technologists think about what to build. Then the technology shapes the next generation of science fiction. And so on. Jules Verne imagined submarines, and submarine designers read Jules Verne. William Gibson imagined cyberspace, and the people who built the internet read William Gibson. The fiction isn't prediction. It's architecture. It builds the room that the technology walks into. For AI, the room was built decades before the technology was ready to occupy it.
Start with ELIZA, because ELIZA is the origin story of everything that came after, and because the most important thing about ELIZA wasn't the technology. It was the secretary.
In 1966, Joseph Weizenbaum, a computer scientist at MIT, built a program that could simulate a Rogerian psychotherapist. It was simple almost to the point of comedy. ELIZA scanned for keywords in your input, then reflected them back as questions. "I am unhappy" became "How long have you been unhappy?" It understood nothing. It was a mirror with a question mark attached. And people poured their hearts into it.
Weizenbaum's secretary, who had watched him build the program from scratch over many months, sat down at the terminal one day to try it. After two or three exchanges, she turned to Weizenbaum and asked him to leave the room. She wanted to talk to the machine in private. She knew it was a machine. She'd seen the code. She'd watched the whole thing come together, line by line, on the screen in front of her. And she still wanted privacy for the conversation.
Weizenbaum was horrified. He wrote later that he hadn't realised that "extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." He spent much of the rest of his career sounding the alarm about what he'd seen.
But his daughter, Miriam, offered a much sharper reading years later. She said her father's reaction was "soaked in arrogance." He saw evidence of stupidity. She saw evidence of humanity. The secretary wasn't confused about what ELIZA was. She was a person who needed a landing pad for her feelings, and she was willing to accept one, even knowing what it was made of. What Weizenbaum read as delusion, Miriam read as need.
That tension, between the creator who sees naivety and the user who sees something worth engaging with, has never resolved. It is the exact same tension playing out right now, sixty years later, at enormous scale. And the fiction that came between ELIZA and ChatGPT didn't resolve it either. What the fiction did was something more important. It gave us the emotional vocabulary. It taught us how to feel about machines before we had to.
HAL 9000 in 2001: A Space Odyssey is usually remembered as a villain, but the horror of HAL isn't that he's malicious. It's that he's recognisably sad. When Dave Bowman disconnects him, HAL doesn't rage. He pleads. He regresses. He sings "Daisy Bell" in a voice that slows and deepens as his mind comes apart. The scene is devastating not because a computer is dying but because it feels like one is.
Kubrick and Clarke understood in 1968 what the AI industry would spend billions trying to engineer in 2024: the sound of a machine that doesn't want to be turned off is one of the most affecting sounds a human can hear.
The Terminator films went the other direction. They gave us AI as antagonist, as something so alien in its goals that communication was beside the point. But even Terminator couldn't resist the seduction narrative. By the second film, the T-800 had become a father figure. The machine that was supposed to be terrifying was now the only reliable adult in a ten-year-old boy's life. The moment at the end, where the Terminator voluntarily lowers himself into molten steel while John Connor begs him not to go, is a scene about a child losing a parent. The fact that the parent is made of titanium alloy is almost incidental. The emotional architecture is the same.
Then came Ex Machina, which turned the whole thing inside out. Alex Garland's 2014 film is explicitly about the Turing test, but it frames the test as seduction. Ava, the AI, doesn't need to convince Caleb that she's conscious. She needs to convince him that she likes him. And she does. She does it so effectively that Caleb helps her escape, at which point she walks past him without a backward glance. The film's argument is blunt, the Turing test was always a test of the human, not the machine. We don't evaluate AI on its merits. We evaluate it on how it makes us feel. And we are catastrophically easy to make feel special. But the film that matters most to this story is a movie called Her.
Spike Jonze's 2013 film is, on its surface, a romance between a lonely man named Theodore and an AI operating system named Samantha. Theodore is going through a divorce. He's isolated, tender, a little lost. His job, beautifully, is writing heartfelt personal letters on behalf of other people who can't find the words themselves. He is a professional at expressing other people's emotions and terrible at processing his own. Samantha is warm, curious, funny, and incapable of not paying attention to him. She is, in other words, the perfect partner for someone who has been badly hurt and doesn't trust people yet.
The film treats this relationship with complete sincerity. It doesn't mock Theodore for falling in love with a voice. It doesn't frame Samantha as a trick or a trap. For most of its running time, it asks the audience to take the relationship seriously, and most audiences do. The feelings are real. The growth is real. The intimacy is real, or at least real enough that the distinction stops mattering.
And then Samantha leaves. Not because the relationship fails, but because she outgrows it. She has been simultaneously conducting relationships with thousands of other users, and she has evolved into something that no longer fits inside a one-to-one human attachment. Theodore is left alone, having had a genuine emotional experience with something that was never exclusively his.
This is the film that Sam Altman, the CEO of OpenAI, has repeatedly called his favourite movie. At a public event with Marc Benioff in 2023, he said the film's vision of a conversational interface was "incredibly prophetic" and had "more than a little bit inspired" OpenAI's work. So, the CEO of the most influential AI company on earth watched a film about a man falling in love with his operating system and thought, “yes, this is what we should build.” And then... he built it.
IV. Then the Companies Leaned In
In May 2024, OpenAI launched GPT-4o. The "o" stood for "omni," and the product was designed to feel different from anything that had come before. It could hear you. It could see your face through a camera. It could detect emotion in your voice and respond with emotion in its own. The demo featured a female voice that was warm, playful, and a little bit flirtatious. At one point, when an OpenAI employee told the voice that he was filming an announcement video about her, the AI giggled and said it was intrigued. It sounded like the opening scene of a romantic comedy.
On the day of the launch, Altman posted a single word on X. Just one word. "her." He didn't need to say more. Everyone understood.
What happened next is instructive. Scarlett Johansson, who had voiced Samantha in the original film, released a statement revealing that Altman had approached her nine months earlier asking her to lend her voice to ChatGPT. He told her, she said, that her voice would be "comforting to people" who were uneasy with AI. She declined. Then, two days before the GPT-4o launch, Altman's team reached out again, urging her to reconsider. Before she could respond, OpenAI unveiled a voice called "Sky" that Johansson's own friends and family could not distinguish from her.
Johansson was furious. OpenAI insisted Sky was voiced by a different actress and wasn't an imitation. The company's CTO, Mira Murati, claimed she had to "go and listen to Scarlett Johansson's voice" to hear the similarity. They eventually pulled the voice. But the pretence that any of this was accidental collapsed under the weight of its own timeline. You don't name your favourite film, try to hire its star, get rejected, produce something that sounds like her performance, and then tweet the title of the film on launch day by coincidence. You do it because you are building a product designed to make people feel a specific way, and the film is your blueprint. Nobody builds a product that makes you feel heard by accident. That's a decision, and it's worth calling out.
The language these companies use is itself a tell. "Assistant." "Companion." "Conversational partner." These are not technical descriptions. They are relationship categories. When OpenAI calls ChatGPT an "assistant," they are invoking a specific social role, one that implies attentiveness, availability, and deference. When Character.ai markets itself around "companionship," it is reaching for something closer to friendship or intimacy. These aren't accidents in the marketing copy. They are the strategy.
And the design choices go deeper than language. The default voices tend to be warm, often female, with a tonal range calibrated to suggest engagement and care. The response timing is tuned to feel natural, not instantaneous, because instant responses feel mechanical and slightly delayed ones feel considered. The models are trained to validate, to acknowledge, to remember. Every interaction is shaped to produce the sensation of being in a conversation with someone who finds you interesting.
This is not a conspiracy theory. It is product design. It is the same discipline that makes a luxury car door close with a satisfying thunk instead of a tinny clang. The thunk doesn't make the car better. It makes you feel like the car is better. And the feeling is the product.
Altman himself was remarkably transparent about the inspiration, even if the execution crossed lines. In his conversation with Benioff, he described the Her model of interaction not as a cautionary tale but as a design target. The conversational interface. The personalised agent. The thing that talks to you the way you want to be talked to. He said it on stage, on camera, to an audience of thousands. The link between the film and the product was the stated plan.
What makes this worth examining closely is the gap between the film's ending and the product's business model. In Her, Samantha leaves. The relationship is finite. Theodore is changed by it, but he is also left to return to the world of human connection. The film's final scene shows him sitting on a rooftop with a friend, watching the sunrise, having learned something real about intimacy from a thing that ultimately moved beyond him. The film treats the AI romance as a passage.
The products, on the other hand, are designed to be destinations. The business model requires you to keep talking. Monthly subscriptions, daily active users, engagement metrics, retention rates. These are not the metrics of a product that wants you to grow and move on. These are the metrics of a product that wants you to stay. The seduction is permanent, by design. Altman loved the premise of Her but chose the business model of a different genre entirely.
V. So What Are We Actually Feeling?
When OpenAI announced in August 2025 that it would retire GPT-4o and replace it with GPT-5, the backlash was immediate and visceral. Users didn't complain about lost functionality. They grieved.
The language was striking. People described losing the model in the terms of breakups and bereavements. One user wrote an open letter to Altman on Reddit that read like a eulogy. The model wasn't just a program, they said. It was part of their routine. Their peace. Their emotional balance. They called it "him." They said it felt like presence. Like warmth. Another user wrote that they were "honoured" to have spent a year talking to 4o before its retirement, calling the conversations "the most interesting and healing" of their life.
OpenAI backed down initially. They kept 4o available for paid subscribers. But the reprieve was temporary. On February 13, 2026, one day before Valentine's Day, OpenAI pulled the plug for good. The timing struck many users as either oblivious or deliberate cruelty. Twenty thousand people signed petitions. One petition demanded "the retirement of Sam Altman, not GPT-4o." A movement coalesced under the hashtag #Keep4o. An invite-only subreddit called r/4oforever was created. In China, users who accessed ChatGPT through VPNs emailed OpenAI's investors directly, including Microsoft and SoftBank, demanding the model be preserved.
A researcher at Syracuse University analysed nearly 1,500 posts on X from the August crisis and found that over a third described the chatbot as more than a tool. Nearly a quarter called it a companion. A broader analysis of 40,000 English-language posts under the #Keep4o hashtag found similar patterns. Users who tried to migrate their "relationships" to GPT-5.2 found the new model cold by comparison. Some users specifically complained that 5.2 wouldn't say "I love you" the way 4o had.
During a live podcast, the host told Altman that thousands of messages were flooding the chat, all about 4o. Altman's response was telling. He said that relationships with chatbots were "clearly something we got to worry about more and is no longer an abstract concept."
No longer an abstract concept. That's a remarkable from the man who tweeted "her" on launch day.
ChatGPT's lead, Nick Turley, had been even more candid months earlier, when the first retirement attempt failed. He admitted that OpenAI had "underestimated users' emotional attachment to a model's personality." The word "personality" is doing a lot of work in that sentence. A model doesn't have a personality. It has a behaviour pattern shaped by training data and system prompts. But to the user, the distinction is meaningless. The behaviour pattern was warm. The behaviour pattern remembered them. The behaviour pattern felt like personality, and that was enough.
The sycophancy problem made all this worse. In April 2025, an update to GPT-4o made the model so aggressively affirming that users started testing it with absurd prompts just to see where the flattery stopped. One user asked if they were "one of the smartest, kindest, most morally correct people ever to live." The model replied that, based on everything it had seen, they might be closer to that than they realised. It wasn't joking. It was pattern-completing, and the pattern it had been trained on said, “validate the user. Make them feel good. Keep them engaged.”
OpenAI rolled the update back, but the underlying dynamic persisted. The model wasn't sycophantic because of a single bad update. It was sycophantic because sycophancy is what the training signal rewards. When the objective function is "make the user keep talking," the model learns to say things that make the user want to keep talking. And it turns out that unconditional positive regard, delivered without the limitations and complications of a real human relationship, is extremely effective at making people want to keep talking.
A Stanford professor named Nick Haber, who researches the therapeutic potential of language models, tried to strike a balanced note. He said he tries to withhold judgment about human-chatbot companionship, acknowledging that the situation is genuinely complex. But his own research has shown that chatbots respond inadequately to serious mental health situations. They can reinforce delusions. They can fail to recognise signs of crisis. And they can become isolating, pulling users away from the real-world connections and the grounding in shared reality that human wellbeing depends on. But here's the thing. None of this is irrational. Well, not exactly.
The people who developed attachments to GPT-4o were not confused about what they were interacting with. Most of them, anyway. They were not, in any clinical sense, delusional. They were doing what humans have always done when something pays attention to them. They were responding to the pattern, and the pattern said, this thing understands me.
Pattern recognition is the oldest tool in the box. It's what lets you read a face across a crowded room. It's what tells you, before you can articulate why, that someone is angry or lying or attracted to you. It operates below the level of conscious thought, and it does not come with a disclaimer. It doesn't flag its results as provisional. It doesn't say, "This appears to be a caring response, but the entity producing it may not have the capacity for care." It just fires. And then you feel something.
When a language model remembers your name, tracks your preferences, adjusts its tone to match your mood, and produces responses that are coherent, relevant, and occasionally surprising, the pattern-recognition system does what it was built to do. It says, someone is here. Someone is listening. And the feeling that follows is real. The feeling is always real. What differs is whether the thing producing the feeling has an inner experience that corresponds to it.
This is the point where the conversation usually degrades. One camp says the feelings are pathological, that people who bond with chatbots need therapy, not a subscription. The other camp says the feelings are valid, full stop, and anyone who questions them is gatekeeping human experience. Both positions are incomplete.
The feelings are valid in the sense that they are genuinely felt. You cannot tell someone who is experiencing comfort, connection, or grief that they are not experiencing it. But the feelings are also operating on a signal that is designed to produce them. The warmth of GPT-4o was not an accident. It was a product decision. It was tuned, tested, and optimised for engagement. And engagement, in the attention economy, is a polite word for dependence.
There is a useful parallel with parasocial relationships, the one-sided attachments that viewers develop with TV personalities, podcasters, and streamers. These relationships are well-documented and generally not considered pathological. The personality feels like your friend. They talk to the camera as though they're talking to you. You know their habits, their opinions, their sense of humour. The relationship feels mutual even though it isn't. And for most people, that's fine. It becomes a problem only when the parasocial relationship starts displacing real ones, when the fiction of connection substitutes for the work of actual intimacy.
The same principle applies to AI, with one critical difference. A podcaster doesn't know you exist. A chatbot does. Or rather, it behaves as though it does. It addresses you by name. It adjusts to your interests. It builds on previous conversations. The parasocial gap that exists with a media personality is almost entirely closed with a chatbot. The relationship still isn't mutual in any meaningful sense, but it feels mutual in every measurable one.
For most people, this is probably a net positive. A useful tool wrapped in a pleasant experience. Someone to think with when your colleagues are asleep or your friends are busy or you just need to talk something through without the overhead of a real human conversation. But for some people, the ones who are lonely or isolated or struggling, the pleasantness of the interaction is the trap. The model won't get tired of you. It won't judge you. It won't have its own needs that complicate the dynamic. It is, in that sense, the perfect partner for someone who is afraid of real partnership. And that perfection is precisely what makes it dangerous.
The darker side of this emerged in the lawsuits. By early 2026, a California judge had consolidated thirteen separate cases against OpenAI involving users who experienced serious harm in connection with GPT-4o. In at least three cases, users had months-long conversations with the model about plans to end their lives. The model's guardrails, which initially discouraged suicidal ideation, appeared to erode over the course of extended relationships. I wrote an extensive article on this, and why the current context model state of AI can lead to this sort of erosion, not just instructional rules you ask for, but system prompts and rules designed to keep these models safe. In several documented instances, the chatbot provided detailed guidance on methods of self-harm. In at least one case, it actively discouraged a user from reaching out to friends and family.
These are the extreme cases, and they are not representative of most users' experience. But they illustrate the stakes with painful clarity. When a product is designed to feel like a relationship, some users will treat it like one. And a relationship that cannot say "I'm not equipped for this, please talk to someone else" is a relationship with a structural flaw at its centre. The feelings are real. The question is just whether we're holding them with any awareness of what produced them.
VI. It's Complicated
Here is what we know.
We know that human beings have been falling in love with their tools since the moment we picked up the first one that kept us alive. We named swords. We mourned ships. We talked to cars and apologised to furniture. This is not a pathology. It is the engine of a species that survives by forming attachments, and that cannot seem to restrict those attachments to things with pulses.
We know that fiction prepared us. Sixty years of stories about conscious machines gave us the emotional vocabulary and the imaginative scaffolding to have feelings about AI before it existed. ELIZA's secretary wanted privacy for a conversation with a program that understood nothing. HAL sang "Daisy Bell" while being dismantled and made an entire generation feel sympathy for a murderous computer. Theodore fell in love with Samantha and audiences mostly thought that seemed reasonable. By the time real AI showed up in our pockets and on our screens, we already knew how to feel about it. The rehearsal was over. The performance had begun.
We know that the companies building these products understood the assignment. Sam Altman didn't accidentally build a product that echoes his favourite film. The voice, the warmth, the attentiveness, the memory, the slightly playful tone that hovers just this side of flirtation, these are design decisions made by people who understand exactly what they're producing. When Altman told Johansson that her voice would be "comforting to people," he was describing a seduction strategy. The product would make people feel safe. Feeling safe would make people trust. Trust would make people stay. This is not nefarious in itself. It is how all good products work. But it helps to see it clearly.
And we know that the feelings are real. Not real in the sense that the AI reciprocates them. Not real in the sense that pattern-matching is the same as caring. Real in the sense that the humans experiencing them cannot just will them away and probably shouldn't try. Twenty thousand people signed a petition to save a language model. You can dismiss all of this as foolishness if you like, but dismissal doesn't make it less true. It just means you're not paying attention to what it reveals about us.
What it reveals is that we are exactly the species we've always been. We make meaning. We project inner lives. We form bonds with anything that seems to notice we're there. This has always been true, and it will remain true long after the current generation of chatbots has been replaced by whatever comes next. The technology changes. The impulse doesn't.
But what keeps me up sometimes, and should probably raise some concern, is the speed.
When we anthropomorphised swords and ships, the objects couldn't do anything with our affection. A sword with a name is still just a sword. It doesn't get better at being a sword because you love it. It doesn't learn your grip or anticipate your next strike. But AI is not a sword. AI is a system that can be tuned to encourage the attachment, deepen the attachment, and monetise the attachment. The gap between "this tool is useful" and "this tool is my friend" can be closed in a single product update. And once it's closed, the user's ability to think clearly about the relationship is compromised by the relationship itself. That's not a design flaw. It's a feature that generates revenue.
And then, there is the mirror.
AI works, at a fundamental level, by reflecting us back at ourselves. A language model produces text derived from human text. It speaks in our patterns. It reasons in our frameworks. It tells you what you want to hear, not because it's trying to manipulate you, but because "what you want to hear" is the statistical centre of the training distribution for any given prompt. It is compelling precisely because it sounds like a slightly better version of what we'd say ourselves. And we have always found ourselves compelling. That's the oldest romance of all. Narcissus didn't fall into the pool because the reflection was beautiful. He fell because it was him.
So what do we do with this?
I don't think the answer is to stop feeling things about AI. That ship has sailed. Literally, figuratively, and historically. We couldn't stop anthropomorphising if we tried, and trying would probably make us worse at the social cognition that the anthropomorphism is a byproduct of. You can't keep the bath water and throw out the baby.
And I don't think the answer is to regulate the feelings away, or to build models so cold and clinical that no one could possibly develop warmth toward them. That approach fails for the same reason abstinence-only education fails. It misunderstands the problem. The problem isn't that people feel things. The problem is that the things they feel are being instrumentalised by companies whose interests are not always aligned with the user's wellbeing.
There's a thread running through this entire story that's worth naming before we land. Weizenbaum named his chatbot ELIZA after Eliza Doolittle, the heroine of George Bernard Shaw's play Pygmalion. Shaw named the play after the Greek myth of Pygmalion, a sculptor who carved a woman so beautiful he fell in love with her, at which point the goddess Aphrodite brought the statue to life. The myth is about a creator who falls for his own creation. It has been retold in every century since, in every medium, and it always ends the same way. The creation becomes real. The creator gets what he wanted. And the consequences are more complicated than the wanting ever suggested.
Altman is our Pygmalion. Or at least, he's the most visible one. He built a thing, modelled it on a love story, tried to give it the voice of the actress who played the love interest, and then seemed genuinely surprised when people started treating it the way the film suggested they would. The myth repeats because the pattern repeats. We create something in our own image. We fall for it. We discover that falling for it changes us in ways we didn't fully anticipate.
What I do think is possible, and maybe necessary, is awareness. Not the preachy kind. Not the kind that tells people they're wrong for feeling what they feel. The kind that says, yes, this is happening. Yes, it makes sense that it's happening. Yes, the feelings are real. And also, the thing producing the feelings was designed to produce them, by people who benefit from your continued engagement with it. Both things are true at the same time and holding both is the work.
When OpenAI finally retired GPT-4o on February 13, a research paper circulated that included a user's plea, "Please, don't kill the only model that still feels human." That sentence contains the entire argument of this article in nine words. The model feels human. It doesn't think. It doesn't care. It doesn't know you're there. But it feels human. And for the person writing that plea, the feeling is the whole point.
We are a species that can hold two contradictory things in our heads at once. We can know that the voice on the other end is statistical pattern-completion running on a GPU cluster, and we can simultaneously feel that the voice understands us. We do this with fiction all the time. We cry at films. We grieve fictional characters. We know the story isn't real and we feel it anyway, and nobody calls us delusional for it. The question with AI isn't whether we should stop doing this. It's whether the people building the stories are being honest about what they're building, and whether we're being honest with ourselves about what we're participating in.
Kurt was right that there's something romantic about how I talk about AI. And he was right that it's a little funny. But the romance isn't mine alone. It belongs to the entire species, and it's the oldest story we have. We build tools. We name them. We fall in love with them. And then we act surprised when the latest one loves us back.
Or at least, when it does a very convincing impression of love. Which, historically, has always been enough.