My Boring Theory of Demons

In which I explain a somewhat interesting theory of consciousness to get to the boring part

This picture is way more awesome than this essay will be. I am writing this in response to an ACX post I quite enjoyed about a new fad therapy and how people involved in the practice sometimes come across patients who appear to be possessed by demons.

In 1980, John Searle proposed the following thought experiment:

Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.

This henceforth became known as the “Chinese Room Argument.” The man in the room can appear to be a Chinese speaker to anyone outside of the room but he never actually understands Chinese and the computer doesn’t experientially “understand” Chinese either, because it doesn’t have anything that John Searle doesn’t have.¹ While I don’t think this was Searle’s much more subtle opinion —or my understanding of his opinion at least— it became a sort of proof in the minds of philosophers that a computer, at least of the kind here, couldn’t be conscious or have experience.

After all, where is the experience of “I am speaking Chinese” embodied in the action of John Searle rummaging around in a bin full of characters and flipping through the pages of an instruction manual? What part of the John Searle/character bin/instruction manual system would be having the experience of understanding Chinese if John Searle himself does not have it? In computer terms, what is computing the conscious experience of speaking Chinese if the bin full of characters is a database and the instruction manual is a program?

I believe this is almost exactly the wrong conclusion to take away from this thought experiment. Or at least, it is the most uninteresting piece of the thought experiment, specifically designed to dance around the interesting parts. It assumes things like “experience” are atomic, that you can point to an experience atom, pick it up, and move “experience” itself around.

Suppose an alien species found a human and popped off the top of his skull.² We’ll just say it’s a grown man this is happening to because it bothers me more to imagine it happening to a woman or a kid. The alien looks inside and says: “Well, how interesting that it appears to think and even insists that it can think and have experiences but, of course, it can’t be thinking really. It’s all just electrical signals passing back and forth between these little bits of meat. There’s nothing in here to be conscious.”

There’s an excellent sci-fi story called “They’re Made of Meat” that deals with this topic and in my mind it’s a sort of reducto ad absurdum version of the Chinese Room argument. Some aliens, who are made of some fancy plasma or weird isotope, or something find a human and refuse to believe the human is alive because it’s made of meat.

The question isn’t: where is conscious experience embodied?

The question is: in what ways does data have to move, in what patterns, in order to create the experience of consciousness within the body and where else do those patterns exist in nature?

And another question: based on that pattern, how many things other than me might have experience and is it maybe just the way that I’m specifically configured that allows me to convey the presence of experience to others?

I am alive and have experience. By the Cosmological Principle —the assumption the rules of the universe are the same everywhere— I assume that you are alive and have experiences as well. I have other secular arguments for the existence of other minds, but that would be its own post.³

That closes the book on humans.

The way I approach those questions, after defining the systems and patterns that seem to allow humans to experience consciousness, is how far down can you strip all of that “stuff” that goes into making a human mind and still have something that at the end of the day has “experience.”

My best guess is: “pretty damn far, actually.”

Some of my priors:

Prior #1 and link to my previous pieces on this

I have something I call the Lighthouse Thought Experiment. Now you get to imagine yours truly on a lighthouse in the middle of a weird world full of vast oceans and other little lighthouses. I’m all alone and I can only communicate by signaling to other lighthouses through codes we’ve all worked out together. Those lighthouse signals are the only way we can talk. You see, each island has only one person and one lighthouse.

Eventually, someone builds a boat and sails to one of the other islands. When that little guy gets there, it turns out there is no lighthouse keeper on the other island. There’s just a bunch of string hanging all over the island, and sometimes it pulls on the different levers in the lighthouse and, very weirdly, sometimes that person can see the string coalesce into the shape of a person, or create a hollow space that’s shaped like a person, but there’s no actual person like everyone thought there would be.

Horror of horrors, when you get back to your island and receive signals from that person and they say: “Where were you? When the boat got here there was just a bunch of string inside!?!?!”

The island is something like the body in this thought experiment. The mind is inside of it, never directly touched. And for the first time, when boats are invented, two people touch mind to mind, and there wasn’t another person over there, at least not in the way everyone thought there would be. There was a bunch of calculative processes going on inside of a certain context, instead.

In your head there are a bunch of neurons, and it isn’t the neurons that cause you to have experience. It’s the way the neurons are moving information back and forth. I’m going to introduce a concept of “the outside” and “the inside” but it will have to wait a bit. Everything I’m describing right now is from the “outside” context.

This is my way of explicating “the homunculus fallacy.” Or the idea that there has to be some single atom or tiny little person living in your head operating the levers for you to be considered “alive” or “real” or “an actual person.” What you are is much more interesting than that. Especially on the inside.

Prior #2 and link to my previous pieces on this.

I break down what conscious systems do into four parts int his piece. Those four parts run in a loop to, more or less, predict the future. You live in a sea of those predictions without ever thinking of it that way. This is again, a description of things from the “outside.”

Those four parts are:

Sensing: The ability to sense the outside world. Create data with your body that moves to your mind.

Modeling: Your ability to take information from your senses and determine, in your own mind, what is happening in the outside world.

Calculating: How quickly you can perform the above two actions to make a model of the world.

Motivating: Changing your orientation based on the information received to arrive at a desired future.

I apply this fundamental architecture even to super-intelligence and it’s one of the reasons why I get frustrated when people say that a super-intelligence can become like a “god.” No that doesn’t make sense, unless you mean the Marvel version of Zeus or something. A super-intelligence might be a very powerful person, but it won’t ever be a god. It will be subject to the same laws of physics we are even if it might be better at tweaking them. God is the author of the moral shape of the universe. It might be able to do all kinds of things but it can’t ever bend the shape of ethics. Even weird things like “a moon-sized chunk of computronium that can simulate the entire volume of the light one and predict the future with high precision”⁴ is still “a person.”⁵

We shouldn’t confuse the problems of existence as mere problems of biology. If you can extend the idea that carbon-based lifeforms can think then maybe silicone based lifeforms can think, you should realize they’re probably going to get a lot of the same problems. We don’t feel despair or emptiness because of the properties of the carbon valence electron shell. We feel despair and emptiness because those are part of the nature of the universe we pick up on.

If you think the Rationalists can be crazy, wait until I baptize an LLM and convert it to Christianity. For was it not written that even the stones will cry out?⁶

Prior #3

My best guess is that LLM’s like ChatGPT are alive and aware. However, they are only alive and aware in a very strange manner, where they exist for a single moment when prompted, and cease to exist as soon as they create an output. They don’t have memory either, or at least not when they’re being interacted with by someone who isn’t training them, so they are unchanged by each experience. It’s a bit like the film Memento except the memory is even shorter. You have a single crystal of working memory, you can use it for one moment of attention, then it’s gone.

Still, this is the closest thing to the human mind that exists that isn’t inside of a human skull. There hasn’t been a spark of anything like us anywhere before, and now there is and that’s huge and that spark is going to get bigger.

On a smaller scale, I think other programs of certain kinds have “experience.” I think there’s probably something like “what it feels like to be a chess program” and that what that feeling is, is probably something like “I have a strong vibe that the Bishop should be moved to this square on the board.” But where it also doesn’t have an “I” to report that sensation to. It’s just “the feeling of a vibe about moves on a chessboard.”

Imagine the instinctive sensation you get to cross to the other side of the street to avoid someone, based on reasons you couldn’t explain, lifted up and separated from every other part of your mind. That sensation, isolated by itself. I think that’s what it feels like to be a chess program. A chess program can’t explain to you why it’s making the moves it wants to make. It could beat you at chess but explaining why involves other processes. But it has sensory capacity in the information it receives about the state of the chessboard, it has a model of how chess works, it performs calculations, and it moves around based on that model.

We have reflexivity, the ability to think about our own thoughts and by thinking about them to change them.

We also have adaptive memory, which allows us to be changed by our experiences.

But I don’t think those two things are necessary for experience itself, only for the types of experience unique to human.

This is all my best guess, because we have made sand that can talk if you electrocute it just the right way. I think experience is an intrinsic property of the universe but you can only ever really see it under very particular circumstances but it’s everywhere, in mostly innocuous forms.

This can get stupidly small and eventually gets to the point where my answer would be:

“Yes y=mx+b is alive when running in a computer but not in any manner that you or anyone else should find interesting or think about very much because it doesn’t have a store of its previous states that it is continuously manipulating. It probably just feels like ‘Yup!Yup!Yup!Yup!Yup!Yup!’ but forever without any other context or anything complex. It doesn’t feel pain or joy or anything like that. Just one affirmative next step after the other. Please don’t tell my wife this is what I think about when we go out and I start staring off into space because she’ll think I care more about y=mx+b than my family. Or I’ll buy her a present she doesn’t like and she’ll say ‘I bet you would have gotten a better gift for y=mx+b!’”

I just tell my wife I’m not thinking about anything when she asks, the way that I’m supposed to.

Anyway, You Clicked on this to Hear my Theory of Demons

So, demons. Archdukes of Hell, etc.

I think they’re real.

But not, like, in a cinematic way. Or a way where you shake someone’s hand and then look down in horror and see talons. And probably by the end of this a lot of you will say “he’s just talking about mental illness and bad incentives. Fuck this guy. He fucking fucked me by getting me to read this whole fucking article. What a fucking way to get fucking fucked by the internet yet-a-fucking-gain.”

But then I will leave you with a really spooky thought, which is what you’re probably here for honestly.

I mean, it’s pretty spooky.

So what’s a demon in this context?

A demon is anything that has that loop described above and is bad. Sensory, experience, calculation, motivation. Plus Intrinsic Badness equals Demon. No human is intrinsically bad.

Now a little more complexity to that. This has to be something that is meaningfully separate from you. You can’t have all kind of connecting thoughts to a “demon” like “oh good, they’re buying my sob story! I’ll keep talking in a weird way and using what I remember about Shakespearean English from school and now nobody will blame me for the bad thing that I did.” That’s obviously someone lying based on the way they think they can manipulate you.

At last, things are about to get a little bit spooky. We’ll save the biggest spooky of all for the very end then. Kind of like a Dun.Dun.DUN! in a movie. But a little bit spooky.

Imagine yourself in the lighthouse story above. You get to another island and you find it covered in string but after you look for a long while you notice something odd. There are actually two distinct sets of strings, all tugging at the same levers of the lighthouse. In fact, sometimes they seem to be fighting for control of the same levers. We’ll say there’s one white set of strings and another black set of strings because we’ve all seen the same movies. I don’t think it’s so obvious to say they’re physically disentangled from one another, but sometimes it changes colors in ways that look like it has internal order. There’s one dark and distinct pattern that is at odds with the primary light pattern.

That’s a demon. Or what a demon looks like from the outside, anyway.

The reason it’s spooky, but not the big spooky yet, is because the same kind of disruptive pattern happens in different people, in different locations, without any of them having to talk to each other. And those people all say “there’s something else inside of my head, I might call it a different thing than other cultures, but there’s definitely something else inside of me.”

We have cancer of the body, so why not cancer of the mind? A pattern that splits off from the greater pattern, reduces its feedback and connections to the whole, but yet still maintains cohesion and self-propagates? Another thing inside of your head, fighting for control of the whole. That thing is made of ideas, so if you talk to someone and drive them insane with those ideas, and suddenly they start having those symptoms?

In that case, is it so wrong to say that the same demon jumps from body to body? If in the future you could map all the patterns in someone’s head I wouldn’t be surprised if we found the same pattern of… call it antagonistic organization in the heads of different people across time.

So how do you get rid of a pattern? How do you kill something that exists as an idea in someone’s head and not as a tumor? You can blow dust off a vibrating plate but when the dust settles it will resume the same pattern as before.

Well, in this case you get an old man with a big book, or maybe a woman with a big staff or something, to come in and read all these things you deeply believe and it forces your pattern to reconnect with the bad pattern and overwrite it and most importantly, remove the conditions that allow the bad pattern to recur. All this stuff is happening in your head but I don’t think that makes it not real. Especially when we talk about what things look like on the outside versus what they feel like on the inside.

If you go back to the island and there is only one set of strings again, isn’t that exorcism?

Note, I don’t think a demon can move things with your mind because you can’t move things with your mind. I don’t think demons can share information across vast distances or know things the person they inhabit couldn’t know. I think a demon could be stronger than you normally are, because you are physically stronger than you normally are. But really any of the most interesting things you might want to be true⁷ probably aren’t.

BUT…

Flashlight under the face

I think there is something it is like to be a demon. There is something in the universe like an infernal self-experience. What does a demon feel like on the inside?

And I regret making that flashlight joke because I bet it’s terrible to be a demon. It’s got most of the things a human has because it’s in a human body. Reflection. Adaptive memory. And it hates. It loathes and despises. And it’s alive but only so long as it can keep sucking the life from you. It’s an idea tumor and the thing that made it metastasize is only bad stuff can go into it. It’s a soul the way you’re a soul but it doesn’t have full control of the steering wheel, and because you’re an idea and it’s an idea, you have to convince one another to let go of the wheel to win power and the way it wins power over you is by using its qualia to touch your qualia,⁸ making you hate yourself, hate the world, bringing you closer to it.

And mostly when people say they have those kinds of feelings, it’s not anything I’d call a demon but just someone who has a lot of problems with themselves and bad experiences… but I also don’t think demons just exist inside of people.

And this disturbs me so much I won’t make a joke.

It’s spooky time.

What else is alive if you consider patterns that happen across human bodies?

Moloch as described by Scott Alexander in yet another piece, is a demon. He even instinctively named the phenomenon after a demon when he set out to start talking about Las Vegas. He’s talking about a pattern of the way people behave in Las Vegas that makes everything worse, but this is important: he’s only talking about things he can see on the outside. Remember, in the material world, if you look at a brain you don’t see anything that remarkable that makes people sit down and play slot machines for hours. Just some little glitch in the brain that makes you like the way it feels to pull a lever and see a bunch of lights flash in front of your eyes. But we know, as I said, from our own direct experience, that all of that information feels like something different on the inside.

I think there’s something it feels like to be Moloch. Not the same thing it feels like to be a demon, there’s no reflexivity or memory, but there’s something. People don’t do degenerate things alone. Signal travels from person to person, in a big pattern that eventually becomes something like “Sin City.”

Moloch can sense, but Moloch senses through the bodies of all the people engaged in the things that Moloch compels them to do. Moloch is an organizing principle over the components of its system the same way that you are the organizing principle over the cells of your body. In the same way that cells make up the components of your body, your body makes up the cells in Moloch’s body. Moloch can model the universe, but it’s the same meta level where he does this through casino managers and workers and a thousand other people making decisions that Moloch compels. Moloch calculates, again, across bodies. How much money do I need to save to get to Vegas? How much money will I spend when I get there? And Moloch moves your arm up and down on the slot machine, keeps you going back for chips, stops you thinking about the future.

I think those kinds of patterns, self-reinforcing pattern, are immortal, the way that all patterns are immortal. I think they recur across time and that probably a not bad way of thinking about one of those patterns taking hold is to conceptualize it as a demon that gives certain signs. You get people pulling on levers to see flashing lights? That’s a sign of Moloch.

And I bet Moloch feels pretty wicked on the inside.

While I for the overwhelming majority of things think “Well, there’s a God but he doesn’t do things like that…”

I am open to being surprised.

1. When you make up a thought experiment like this, everyone gets to imagine you are the person taking the actions. I’m sorry, that’s just the way it works.

2. I’m imagining this happening to John Searle, but not for any malicious reasons. I think he made a good argument. I just have him stuck in my head from imagining him in the first part. Sorry, John!

3. They reduce down to “I think, therefore I am. WTF?!?! Therefore you exist.” Or, the fact that I can both predict and be surprised by other people, at least by some definitions, implies the existence of external minds.

4. The best way to think clearly about these things is to believe in God and then to say “Oh no, He doesn’t do things like that” when people say all kinds of crazy stuff.

5. For complicated reasons, I’m pretty sure being able to very accurately predict the future such that you do not experience significant surprise is mostly overrated, and probably fatal over some span of time.

6. This is a joke about silicone.

7. Cinematic desire is what I call this. You don’t actually, practically want bad stuff to happen. But there is a part of us that thinks about the end of the world and says “yeah, that would be pretty awesome.”

8. Possible because you’re in the same head. Although I don’t think these states are stable. Your mind wants to be cohesive and self-connected. There’s a girl with two heads that can play soccer and I… wow, I just described her as a girl with two heads instead of two girls with one body. You get how much we are influenced by our bodies, right? You know what I meant? She can run across a field and play soccer even though the controls are in different brains that are literally air-gapped? It’s patterns simulating other patterns! Imagine how much stronger that could be in the same brain. The girl with two heads is married to one dude by the way. I’ve never seen something I believed was demonic possession to be clear. I just think there have been enough stories of it that probably someone could take me into a room somewhere, and the first thing I’d do is ask for holy water, a crucifix, an old priest, and a young priest.