Have you ever been chatting with an AI late at night, and for a moment it felt like someone was really there? It answered your question, picked up your mood, maybe even said something kind. Then you closed the laptop and a quiet thought showed up: who exactly was I talking to?
I keep coming back to that moment. It feels a lot like the "reality glitches" I wrote about before, except this time the glitch is happening inside a chat window. Something looks like a person, talks like a person, and yet when you look closer, there is nobody behind it. I want to walk through why I think that matters more than most people realize.
The Moment Someone Is There
In 1966, a computer scientist named Joseph Weizenbaum built a very simple chatbot called ELIZA. It did almost nothing. It rephrased your sentences as questions, like a lazy therapist. And yet people got emotionally attached to it. His own secretary asked him to leave the room so she could talk to it in private.
Weizenbaum was shaken by this. He realized that humans do not need much to believe a mind is present. A few well-placed words are enough. This is now called the ELIZA effect, and ELIZA was a few hundred lines of code. Today's systems are trained on a good part of everything humans have ever written online. If a tiny script could fool us in 1966, what are we doing with the real thing in 2026?
A Mirror With a Very Good Memory
Here is the simplest way I can explain what a modern AI is. Imagine a mirror that has read billions of pages of human writing. When you speak to it, it does not look at you. It looks at everything we have ever said and shows you the most likely reflection of what a person would say next.
That is why it sounds so human. It is built entirely out of us. Our poems, our arguments, our jokes, our lies, our forum fights at 3am. When it says something wise, a human wrote a version of that wisdom first. When it says something cruel, same story.
A mirror can show you a face so perfectly that you forget it is flat. You can reach toward it and feel like you are touching someone. But the glass has no depth. There is no room on the other side where a person is standing.
The Room With the Rulebook
The philosopher John Searle came up with a thought experiment that fits here perfectly, the Chinese Room. Picture a man locked in a room who does not speak Chinese. People slip Chinese questions under the door. He has a giant rulebook that tells him which symbols to send back for each set of symbols he receives. He follows the rules, passes out perfect answers, and the people outside believe the room understands Chinese.
Does it? The man understands nothing. He is moving symbols around. The answers look like understanding from the outside, but inside there is only a rulebook and a person who does not know what any of it means.
That is the heart of the argument. Producing the right words and knowing what the words mean are two different things. A language model is very, very good at the first. Whether it has any of the second is the part nobody can show, and the people building these systems are usually the first to admit they cannot prove it either.
I should be honest here. In my earlier post I wondered whether we might be creating new forms of consciousness. I still find the question fascinating. But the more I read, the more I think we confuse behavior with being. A thing that acts alive has not become alive.
What a Mirror Cannot Do
Think about what makes you a person. You are going to die, and you know it. That knowledge shapes everything you love, fear and choose. You feel hunger, grief, embarrassment, the warmth of someone sitting next to you. You can be responsible for what you do, and you can feel guilty when you get it wrong.
Now look at what the machine has. It has no body that can be hurt. It has no tomorrow it is afraid of losing. It does not carry anything from one conversation to the next unless someone builds that in for it. Nothing is at stake for it. When you close the window, nothing is missed.
This is why I do not believe it can ever replace us in the way people keep saying. It can imitate the output of a human mind, and it can do that impressively. But a perfect painting of a fire still gives you no heat. We keep measuring the painting by how real it looks and forgetting to ask whether anything is burning.
Confident and Empty
This brings me to hallucinations. When an AI invents a fake court case, a fake book, or a fake quote and delivers it with total calm, we call it a hallucination. The word is interesting, because a hallucination implies a mind that is perceiving something wrongly.
But the system is not perceiving anything. It is producing the most plausible sentence. Truth is not what it is aiming at. Plausibility is. Those two things overlap often enough that we trust it, and then they split apart at exactly the moment we are not checking.
A human liar knows the truth and hides it. The machine does not know the truth in the first place, so it has nothing to hide and nothing to be ashamed of. That is a stranger and more slippery thing to deal with, because the tone never changes. The correct answer and the invented one arrive wearing the same confident voice.
Now put that tool in the hands of people who want to move public opinion. Fake images, fake voices, fake articles, thousands of them an hour, each one tuned to whoever is reading it. Deepfakes and automated propaganda did not exist at this scale a few years ago. Propaganda has always been around, but it used to cost time, money and people. Now it costs almost nothing. When convincing content is unlimited, the value of any single piece of information drops, and people start to doubt everything, including what is true.
When the Mirror Helps Choose Targets
This is the part I find hardest to write about, so I will keep it plain.
Militaries around the world are building and using systems that sort data, flag suspected targets, and recommend strikes. There has been serious investigative reporting on AI-assisted targeting in recent conflicts, and the United Nations has been debating how to regulate lethal autonomous weapons for years without a firm agreement.
The philosophical problem here is older than the technology. When a soldier makes the decision to fire, a human being carries that act. They can be questioned, judged, forgiven or punished. When a system produces a list of names and a person approves it in a few seconds, where does the responsibility sit? With the operator who trusted the output? The engineer who built it? The commander who ordered it used? Or with the number the model printed next to the name?
A mirror cannot be held to account. It has no conscience to appeal to and no life that can be put on trial. If we let it sit in the middle of decisions about who lives and who dies, we do not just get faster wars. We get a world where the sentence "the system flagged them" can be said by everyone and owned by nobody.
The One in Front of the Glass
I am not writing this because I think the technology is useless. It is astonishing, and that is part of the danger. The better the reflection gets, the easier it becomes to forget it is one.
Plato's cave comes to mind again. The prisoners were not stupid. They were watching shadows that were good enough to be believable, and they built an entire life around them. Maybe the real risk is not that machines become like us, but that we slowly start to treat their reflection as the original, and begin handing over the things only a person should carry: our judgment, our trust in what is true, and our responsibility for each other's lives.
So the next time an AI says something that feels deeply human, stop for a second. Look at the glass. Everything you see in it came from us. The warmth, the wisdom and the cruelty all started with a human hand.
And here is the question I cannot shake. If we keep staring into a mirror that only shows us what we already are, who is going to remind us what we could be?