What you’re saying Lee, makes me wonder — if there’s a system within the brain that turns “brain noise” into “brain information”, then couldn’t the brain thus directly be classified as “an information processor”? And regardless of that question, I wonder what brain system you’re proposing gets this done?
I can see what you’re saying about “lynch-pin of structure” in an epistemic sense Lee. I get the concept. Mind you that in an ontological sense I consider all of causality tied together as a single system which requires each part for the structure to be whole, so one can’t be more important than another. In the end it should all be the same unified thing, or causality itself. But given this qualification, sure, there are things that seem more fundamental to us.
Regarding a brain lynch-pin itself, I think it’s pretty well established that the synaptically gated firing of neurons resides as a processing structure for input information. Here the brain may be considered a massively parallel computer. Essentially input information (such as from a thumb whack) gets algorithmically processed. Then the processed information goes on to do various things, such as release hormones and operate muscles. There should be nothing semantic here yet and so it should essentially be like one of our computers operating a robot. This perspective suggests that there must be a loop in brain function which creates the value dynamic that we feel, such as when a thumb gets damaged. So I presume the processed information informs something causally appropriate to exist, for example, as the experiencer of a thumb whack. The only thing that I know of that might serve this purpose is the electromagnetic field associated with the correct sort of neural firing. But I wonder if you’re going to propose a second reasonable possibility? Or instead are you going to challenge my position that the synaptically gated firing of neurons may usefully be said to reside as non-semantic information processing?
I understand that computational and functional perspectives aren’t everyone’s cup of tea. My newsletter, especially the recent series on information — and this new one on meaning — lean on those frameworks for two reasons:
1. I am a computational neuroscientist, so I am naturally going to draw on my areas of expertise. Computationalism is one side. Biology is the other half of my training. These two sides don't always fit neatly together. The tension between them can spark useful questions.
2. Computational functionalism is one of today’s most influential views, so I think it deserves a fair look — at both its strengths and its weaknesses.
That said, all viewpoints are welcome here. Like you, many readers do not subscribe to functionalism. And I’m happy (within time constraints) to try to understand their views. As I see it, diverse perspectives only enrich the conversation.
Would it help if we separate “information” from “meaning”? If we use information in the Shannon sense, then we might agree that the brain processes loads of information (in the Shannon sense). The question then turns to — how that information becomes meaningful.
Hi! You raise a genuinely important issue about where meaning comes from.
There is a lot of confusion around this term “information.”
In everyday language, information already means something (semantics is baked in). But in information theory (and in fields like neuroscience), information is simply structured variation that reduces uncertainty. It’s about patterns and correlations, even before anyone interprets them.
Imagine a temperature sensor hooked up to a thermostat. The sensor spits out a voltage proportional to the outside temperature. By itself, that voltage isn’t “about” anything in a human, semantic sense—but it still carries Shannon information, because tighter voltages map onto narrower ranges of possible temperatures. The thermostat can act on that pattern with zero need to “understand” what ° Celsius means.
This double use of the word information creates a lot of confusion.
Especially, when scientists say the brain is an information-processing system. When neuroscientists call the brain an information-processing system, they mostly mean it transforms, compresses, and routes those raw, pattern-laden signals so an organism can perceive, learn, and act. And you're absolutely right that as explained, this transformation alone doesn’t necessarily explain meaning.
You put your finger on the key question right at the end: what kind of system is needed to construct meaningful information from raw input? That’s exactly where I hope to take this series in the coming weeks. Looking forward to reading your thoughts.
Pardon my interjection, but I've noticed that for some reason First Cause can be breathtakingly rude and arrogant. I'm not sure he's aware how offputting it can be, but it's time someone mentioned it. In hopes of adding some levity, I can only recommend Fry and Laurie's "American Ass sketch." (https://www.youtube.com/watch?v=N8qw3oWvDpo)
I'm sorry you feel that way, Lee. I saw that you had written some earlier comments, but it looks like you’ve since deleted them. I was planning to respond -- you had raised some interesting points.
A very good intro to the Chinese Room argument, and the common criticisms against it. I think this thought experiment is the epitome of what's wrong with taking philosophical thought experiments as authoritative in the same manner that actual experiments are. Searle's argument is rhetoric for a certain set of intuitions.
On the Wrong Model reply, it's worth pointing out that any artificial neural network you've used to date was implemented with a symbolic program. Neuromorphic computing might change that, but right now, ANNs are built on a symbolic foundation. IIT advocates use this fact to argue that they're still the wrong kind of causal structure.
But to Searle's reply, none of his neurons understand English. In fact, without Broca's area and Wernicke's area, neither does his brain. Yet with all the regions working together, we say he does understand it, which is the System's Reply used for his brain.
But the biggest issue I've always had with this thought experiment is the amount of time it would take Searle to reply to any question. Responding to even the simplest question would likely involve following billions, if not hundreds of billions of instructions. If we assume Searle can perform one a second, and doesn't take sleep, meal, or bathroom breaks, he might finish his first billion instructions in about 30 years. We can make him more productive by wheeling in a computer to help him, but now the original intuition of the thought experiment is weakened, or it should be.
To me, the best thing that can be said about the Chinese Room is it spurs discussion. But what it really does is clarify people's intuitions.
A computation taking one second on a machine or two million years in the "Room" are equivalent as long as the same steps are executed in the same order, aren't they?
They are. Although it's worth noting that relative to the environment, the one second version, assuming it has the right outputs, is much easier to intuit as comprising understanding.
The whole point of the thought experiment is to drive the intuition that understanding can't possibly be there. But millenia or eons of manual activity compressed into seconds? Maybe that doesn't change the intuition for some, but it does for me. (Not that I think trusting intuitions in this area is productive anyway.)
Yes, the Chinese Room, like many other thought experiments, is “rhetoric for a certain set of intuitions.” Many famous thought experiments — like Mary’s Room or the Trolley Problem — also trade on intuition.
I think this hits on something interesting: what are thought experiments that trade on intuition good for, if anything? If they help highlight our intuitions, that could actually be a strength, right? They bring those intuitions out into the open. From there, we can ask: do our intuitions hold up when we test them against empirical evidence?
I like the phrase Dennett uses — intuition pump. The Chinese Room thought experiment gets us to feel the intuition that understanding must be more than symbol manipulation. But to test whether that intuition is correct, we’d need to bring in logic and/or empirical work. Maybe that’s where the mistake happens — assuming intuition tells us something true.
I guess that’s what you meant by: “the best thing that can be said about the Chinese Room is it spurs discussion. But what it really does is clarify people's intuitions.” That feels spot on.
Good point about the Wrong Model reply, too. LLMs are trained on symbolic text and often operate on discrete tokens at the first layer, so they’re not exactly symbol-free. If we think a system that computes over symbols can’t understand, wouldn’t a system that computes over vectors and matrices face the same problem? Neuromorphic computing is an interesting twist. Though, I could imagine someone arguing that if it’s built the same way our current models are built, it would still run into the same problem.
Right. I don't want to give the impression that I think thought experiments are useless. But we should remember in which way they're useful, not as arbiters of what reality is, but in flushing out our intuitions. In my view, the best ones show the limits of those intuitions, where they they break down or become contradictory.
But I'll admit to having a particular attitude toward Searle's. Maybe it's Searle's acerbic delivery, or the way I frequently encounter people who only seem to take account of his side, and not the many responses.
But most likely it's just the reliance on intuition as a guide to reality. It seems like the history of science shows that intuitions can be useful starting points for an investigation, but not as anything authoritative.
Haha, yes—the famous John Searle snark! I remembered he was frank, but re-reading his classic papers for this essay, I was reminded again how sharp-edged his tone really is. He peppers his argument with swipes at other views, and after a while, I started to wonder what’s driving that. It’s funny how sometimes the more confidently someone presses their point, the less confident we become in their claims.
Outstanding article. I understood the Chinese room argument conceptually before, but never really broke it down like this, and this brings it into question in my mind. Thank you for your writing.
Nicely done! Succinct, comprehensive and short introduction to a fascinating debate I was around to witness as a young man. I’m looking forward to this series and glad you’re back. All the best, John.
Thanks, John! It’ll be great to get your take on whether the debate has really moved beyond the original talking points — or if we’re still hashing out the same old puzzles.
Awesome Suzi! It’s probably well known here already that I agree with Searle. In 2019 however I felt like I needed to develop a thought experiment that in some ways goes further than his Chinese room. Instead of the vague concept of “understanding” that functionalists can easily claim is possessed by a hypothetical future computer that passes the Turing test for several months, I use the distinct concept of “thumb pain” that we should all grasp similarly. Furthermore I identify the problem of computational functionalism to be one of failing to complete the “output” component required for any input (like a key stroke), to effectively do anything. Should an experiencer of thumb pain exist when paper with the correct marks on it gets algorithmically processed to create more paper with the correct other marks on it? Of course not! To be causal the output paper should need to inform something causally appropriate that would itself reside as such an experiencer. Not only does this go beyond Searle’s case in that regard, but I even float the possibility that processed brain information goes on to inform a neurally produced electromagnetic field as the general substrate of consciousness. Then Google’s Notebook LM program took the text of my post and created an incredible podcast that breaks down my argument into more comprehensible plain speech units. How could this AI have enhanced what I’d written so much with just a couple minutes of processing, when if humanly produced it would have cost big money? Today they give away algorithms for free that are good enough to trick us into believing we’re dealing with people!
Ah, good—yes! There are some interesting arguments in the literature suggesting we should distinguish between the link between understanding and language, and the link between understanding and other experiences like pain, moods, and bodily responses. That might be an important distinction. But I'll get to that in an upcoming essay.
For now, I want to turn back to an earlier conversation we were having. I want to make sure I fully understand your view, because I’m still not getting something, and I'm not sure what I'm missing.
If I’m following: where functionalism says consciousness could in principle be implemented in any substrate that has the right causal structure, you’re saying—no, the substrate itself matters critically. Is that right?
And in your view, that critical substrate is EM fields. You’re saying EM fields are consciousness. So when you talk about information needing to “inform something causally appropriate,” the causally appropriate thing is the EM field. Yes?
Here’s where I get a bit stuck. In other comments, you’ve suggested that we could test your idea by generating weak EM fields that mimic those typical of neural firing, without those fields altering the firing of any neurons. But in that case, wouldn’t the EM field have no causal power? It’s not doing anything to change the system’s behaviour, right?
So I’m curious: how does your view avoid running into the same kind of problem that epiphenomenalism faces—where you’ve got a conscious field that exists but doesn’t cause anything?
Great questions Suzi! The last one is the big one so hopefully we can straighten this out. I’ll take them in order.
“…where functionalism says consciousness could in principle be implemented in any substrate that has the right causal structure…”
I’d alter this a bit because as I understand it computational functionalists don’t believe there can be “a consciousness substrate”, and especially when they also call themselves “illusionists”. What you might have meant to say there was something more like “consciousness could in principle be implemented by means of any computer with sufficient processing capacity”? And actually we’re in agreement there. I’d say the difference between us is that perceptions of consciousness is what they focus on while I go deeper to a potential causal substrate. Thus they might posit that thumb pain will reside by means of processed information in itself (and so no substrate that’s informed by it), while I posit that processed information can only exist as such to the extent that it informs something causally appropriate (or a potential consciousness substrate).
“And in your view, that critical substrate is EM fields.”
I’d say something more nuanced like “While I can see how processed brain information might be informing a neurally produced electromagnetic field to exist as consciousness, from my own grasp of brain function I’m not aware of a second reasonable possibility.”
“In other comments, you’ve suggested that we could test your idea by generating weak EM fields that mimic those typical of neural firing, without those fields altering the firing of any neurons. But in that case, wouldn’t the EM field have no causal power? It’s not doing anything to change the system’s behaviour, right?”
That’s the true question that I think’s been keeping you, and maybe various others, from grasping my position. So if we can now sort this out then I should be able to better clarify the matter in my next post.
Let’s say we induce exogenous field energies in someone’s brain that thus alters their neuron firing, as in the case of Transcranial Magnetic Stimulation. If this also alters someone’s consciousness then we shouldn’t know if it was the altered firing that created this effect in an unknown way, or rather the altered field itself was the consciousness that the exogenous energy interfered with. And regardless, because brains don’t produce energies anywhere near TMS strength anyway, I don’t know how one might effectively argue that consciousness exists by means of those sorts of energies that aren’t found by means of endogenous brain function.
Let’s now consider the opposite approach. We implant transmitters in various interesting parts of the brain that produce energies which ought to constructively and destructively alter the EM field associated with typical synchronous neuron firing, though shouldn’t be strong enough to affect brain function directly. Furthermore let’s say that a test subject’s experience of vision resides in the form of EMF associated with the correct synchronous neural firing. If (1) a transmitter close enough to the neuron firing that creates the field that’s experienced as vision (2) were to produce an energy that constructively and/or destructively alters the vision EMF, then (3) vision ought to become distorted in various ways. Here the EMF experiencer who knows they’re suppose to report such disturbances, might indeed correspond. From this theory the field/experiencer ephaptically couples with motor neurons that send signals to speech muscles for such report. Therefore that’s my plan — hopefully specialists would say that the exogenous energies shouldn’t be strong enough in themselves to directly affect neural function, but if EMF consciousness itself does become affected then apparently this larger energy structure (the thinker, theoretically) would go on to affect neural function for standard oral report. If these exogenous energies were reliably reported to distort someone’s consciousness, or could eventually even be tailored to give someone a specific type of experience, then what would a second explanation for this be?
Eric, I’ll reply to you here as well, as you referenced Suzi and your response to her. Suzi is pointing to the problem of epiphenomenalism here, and I think you’re not appropriately adressing it! Let me attempt to use your own kind of argument to show this.
You’ve been clear in our previous exchanges that the EMF theory is not a “strong emergence” theory, and that it respects the laws of physics as we know them on Earth. You posit the EMF as the 'causally appropriate thing' that processed information informs.
So, let's take that premise to its logical conclusion. Imagine we use 'marks on paper,' as you say, to perfectly compute Earth's entire physical evolution from a past state, say at some point in 2018, to this very moment in 2025. This grand calculation, strictly obeying all known physical laws, would encompass every atom, every force, and crucially, all electromagnetic fields in their entirety, including those generated by brains like yours. This comprehensive physical simulation would, therefore, chart the precise sequence of events leading to you becoming a proponent of EMF theories, your development of the thumb pain and marks-on-paper examples, every employment of that example, your writing of this entire post. It would also account for every interaction you and I have had, as well as my own journey to computational functionalism.
If these calculations—these 'marks on paper' representing the complete, causally closed physical system of Earth—show how the physical EMF (generated by neural activity and interacting according to Maxwell's equations and quantum mechanics) fully accounts for the neural states that lead to you thinking, writing, and arguing that this EMF is the seat of conscious experience, then what specific, additional causal work is being done by a 'non-computational' or 'non-functional' property of this EMF? If all the effects of the EMF, including your arguments about it, are already predicted and explained by its computable physical interactions within this simulation, then any proposed 'experience' or 'causally appropriate informed status' beyond these physically determined effects becomes indistinguishable from an epiphenomenon—a consequence, perhaps, but not an independent cause of your arguments or any other physical event.
Within the constraints of physical laws, whatever they may be, the ‘marks on paper’ argument undermines and ultimately refutes itself.
To tidy up the confusion around functionalism. Functionalism holds that any physical system that instantiates the right causal (in practice, computational) organisation would realise the same conscious states. That’s substrate-independence, not substrate-absence: some physical stuff must still be doing the work.
Illusionists then argue that what needs explaining is why those physical/computational processes lead us to judge that we have ineffable feels. They don’t deny the neural substrate; they deny that the ‘feels’ are anything over and above it.
Now, onto the interesting ideas...
To make sure I’m understanding you correctly, I’ll break it down as I see it—you can let me know where I’ve misunderstood you.
1. I think we’re on the same page about raw field strength
A figure-8 TMS pulses top out around 1.5 – 2 T at the coil face and are strong enough to depolarise cortical axons, eliciting phosphenes or motor twitches depending on the target site. Endogenous brain fields live many orders of magnitude lower—roughly 5 – 100 nT for magnetism and < 5 mV mm⁻¹ for electric gradients.
So yes, there’s a huge power gap.
2. But as I understand it, field-strength isn’t what EM-consciousness theories lean on
McFadden, Pockett and others locate conscious content in the spatiotemporal patterns of those weak endogenous fields, not in brute amplitude. A giant, homogeneous TMS pulse seems like the wrong pattern.
3. Can we really not tell whether TMS changes consciousness via neurons or “the field itself”?
I think we can. There have been a number of papers that show this. The experiments can be a bit tricky to read, but essentially, the experiments record both brain activity and reports and they show that tACS or patterned TMS first entrain neuronal firing; the subjective effects (e.g., flickering phosphenes, phase-dependent brightness shifts) track the phase of that entrainment, not the raw field amplitude. Somatosensory input can be blocked and the entrainment persists, pointing to a direct neuron–field interaction rather than a free-floating field phenomenology.
4. Why the “silent-field” transmitter is a physics long-shot
Any external source powerful enough to reshuffle the brain’s endogenous pattern must, by Maxwell’s laws, induce charge movement and nudge membrane potentials. Slice and modelling work shows that < 1 mV mm⁻¹—the very regime needed to re-paint the pattern—already shifts spike timing and network synchrony.
In other words: if you can sculpt the field, you’ve already touched the neurons.
5. What alternative explains a reported distortion?
Plain-vanilla neural entrainment seems to cover it: the transmitter perturbs local spikes (directly or ephaptically), those spikes ripple through the circuit, and the subject reports a change. I don't see a need to reify the field as the experiencer.
So, the way I see it:
Field theories remain interesting — weak endogenous EM patterns may well modulate, coordinate or even help bind neural activity. But current evidence seems to suggest that the field does its work through the neurons that create it. If a future experiment can imprint a complex conscious scene by reshaping the field while leaving spikes untouched... well, that would indeed be revolutionary. Until then, Occam's razor seems like the best approach. And because “neurons first, fields as influence” remains the simpler — and better-tested — story, it seems like the most reasonable working hypothesis to me.
Thanks for so diligently considering my testing proposal Suzi! This should help improve my next essay. I’ll start with your “4. Why the “silent-field” transmitter is a physics long-shot”.
To help me assess this question I went back through McFadden’s first published paper on the matter, dated January 2002. (https://philpapers.org/archive/MCFSFA.pdf). Indeed, the suggestion there was also that EM fields always have at least some potential effects on neurons that are close enough to the source. It even mentioned that extremely small field effects from the firing of just one neuron can have some potential field effects on approximately 200 neighboring neurons. Instead I was under the impression that neurons are somewhat protected from at least minor field effects given their myelin sheathing — apparently that just helps create faster ionic transfers. I guess it’s mainly the computers we build that require shielding from EMF energies. Still the sensitivity of neural function to EM influences does at least suggest a greater potential role for the ephaptic coupling that EMF consciousness would require for information to loop from the field back to the brain. The net effect as I understand it is that the field effects of implanted transmitters should cause certain neurons that are on the edge of firing anyway, to be more likely or even less likely to do so.
Anyway I was wrong to hope that my proposed test would be reasonably clean in this sense. Thus if subjects were to report strange consciousness effects by means of appropriate exogenous transmissions from within their heads, then scientists would still need to determine if it’s because of field effects or rather neural effects? It’s hard to quite know how they might get around this issue until actual experiments are done. Would they learn to use exogenous EM energies to give people novel phenomenal experiences in the form of distinct images or sounds? In that case it would seem more likely that they were directly altering a subject’s consciousness rather than indirectly doing so by means of neural firing alterations that create such effects.
Furthermore observe that there’s another way such testing might go. When exogenous energies are generated that ought to alter the field associated with ongoing synchronous neural firing, it could be that subjects would not notice or report the occurrence of strange phenomenal circumstances. With enough such results, McFadden’s theory would then need to be dismissed. I’m not currently aware of any truly falsified consciousness proposals to date. I presume this is because beyond his, consciousness theories seem unfalsifiable in general. Do you know of any consciousness theories that have been empirically disproven, or like McFadden’s, could potentially be?
Actually beyond your point 3 that it ought to be possible to understand if TMS alters consciousness by means of neural effects or direct EMF effects, McFadden’s paper also suggests such a possibility. At one point he mentioned TMS which incites neural function that doesn’t exactly stop when the exogenous energies stop, though rather milliseconds later. Here it was presumed that exogenous transmissions can cause chains of neural firing that thus continue a bit longer. This tendency might help scientists with my proposed test as well. If phenomenal effects from transmitters in the head are reported to end exactly when transmissions end, and perhaps corroborated with monitoring like EEG, then this would be a sign that consciousness was altered directly by means of exogenous field effects. So that would help his case. But if phenomenal effects from transmitters in the head were found to continue on for milliseconds later, then this would suggest the incitement of endogenous neural firing. That should at least not help his case.
On functionalism mandating computational substrate independence, yes I agree. And indeed, I’m a functionalist in that sense too — I don’t think it should matter what substrate does the information processing. We differ regarding what comes next however. I believe that information should only be said to exist as such to the extent that it goes on to inform something causally appropriate. Thus pending evidence, processed brain information might be found to inform a neurally produced electromagnetic field which itself exists as what sees, hears, thinks, and so on. Functionalists however do not entertain substrate based consciousness proposals as I understand it, thus mandating an inherently unfalsifiable position. Furthermore it should be difficult for them to voluntarily postulate that consciousness might required a causal substrate because then they’d lose various long cherished sci-fi implications, such as a coming need for robots to be given human rights, a coming “singularity” where they become far more intelligent than us and so take over our world, and that humans might someday shed their biology for their consciousness to reside by means of technological machine function.
I consider illusionists essentially the same, but see them more as the attack faction in the sense that they like to tell people about the supernatural components to their various conceptions of consciousness. We’re aligned on that, though I go one step further to observe the spookiness of their substrate-absence based conception of consciousness. Yes the computer is considered to have substrate, though the consciousness it creates is proposed to exist as nothing in itself.
No worries, Eric! Like you, I love these sorts of conversations. 😊
On your point about myelin — yes, myelin speeds up the transmission of electrical signals. It does this by preventing the loss of electrical current across the membrane, allowing for faster conduction along long axons.
But in relation to your ideas: while myelin insulates axons, the bulk of the electric fields that form local field potentials actually arise in unmyelinated dendrites and somata in the cortex. So myelin doesn’t really “shield” the structures most involved in generating or sensing endogenous fields.
The standard view on EM interactions seems to have shifted over the years. Ephaptic effects — where electric fields influence nearby neurons — are now fairly uncontroversial and are getting more attention in research. But most measured effects are still weak (<1 mV) and highly local.
That said, ephaptic coupling isn’t always that tiny. While classic cortical studies reported <0.5 mV shifts, recent work on the cerebellum has shown multi-millivolt fields strong enough to pause the firing of dozens of nearby Purkinje cells up to ~60 µm away. These are powerful micro-scale synchronisers — but still a far cry from the kind of brain-wide, content-rich field a McFadden-style theory would require.
It’s super interesting work, though. There’s a lot to explore in this space!
As for invasive stimulation devices (like depth electrodes delivering oscillatory electric fields) — yes, they clearly modulate perception. But so far, the data suggests that these effects originate from synaptic-network activity, not some “silent-field” route. No one has demonstrated a clean dissociation. Everything seems tightly coupled, and the causal arrows all point to neural activity.
You also mentioned that TMS after-effects persist for milliseconds after the pulse. Definitely true. A single TMS pulse can evoke bursts of cortical firing beginning 1–4 ms after stimulation, and those reverberations can last tens to hundreds of milliseconds. Behavioural and EEG effects tend to track that neural activity.
These studies are hard to run! I had a rough time trying to record simultaneous TMS or tDCS with EEG. The very same pulses or currents you use to stimulate the brain swamp the EEG electrodes with artefact. So we had to spend a lot of time designing hardware and analysis pipelines just to recover the actual brain signals underneath the noise.
For more on that front, here's a link to a Frontiers Research Topic from 2017—11 papers on the causal role of neural oscillations in perception and cognition:
On whether consciousness theories can be falsified, I think you raise an important point. Many are, unfortunately, not. But I’d argue that the Orch-OR microtubule model is an exception — and that it has been seriously undermined. The calculated decoherence times in the brain are far shorter than the theory requires, making its central mechanism physically unworkable.
As for functionalism, some would say it is falsifiable. It posits multiple realisability, but makes testable predictions: if two systems share the same causal organisation, they should exhibit the same cognitive or behavioural profile. It could be challenged by finding a system where causal structure is preserved but cognition/behaviour clearly differs. So I’d say functionalism remains empirically open — not automatically unfalsifiable, though of course, finding a decisive test is notoriously tricky.
And finally, illusionism. I think a lot of people misread what illusionists are actually claiming. It can sound like they’re saying we don’t have experiences at all — but that’s not it. Thinkers like Dennett and Frankish argue that we do have experiences — it's just that our introspection leads us to believe they are something they are not.
Some of your points seem particularly hopeful to me Suzi! So specialists are already implanting depth brain electrodes that deliver oscillatory electric fields? For my proposed test to even potentially be attempted, I’ve been worried that unique new brain transmitters would need to be invented. Of course the purpose of these current instruments must be to alter neural firing given that few today suspect consciousness itself might exist under the brain’s EMF. Hopefully such instruments could be calibrated to simulate the synchronous firing of specific numbers of neurons, maybe in the tens, hundreds, or thousands? Ideally this could even be varied on the fly just as individual neuron firing can vary on the fly. Mind you that a successful test wouldn’t necessarily have to be “silent” to neural effects. Directly affecting neurons would simply add the uncertainty of whether a given documented consciousness alteration was neural or field sourced. And your point about myelin EMF protection could be helpful as well since a given depth transmitter might be implanted in an area where neurons are more heavily myelinated and thus a bit more protected from a transmitter’s influences.
There’s something else that you said which might ultimately make such testing quite feasible. If there tends to be a 1 – 4 ms phenomenal alteration delay after stimulation, with effects lasting for tens to hundreds of ms after a given TMS pulse ends, then shouldn’t this serve as a signature of neural tampering? Conversely notice that the light speed function of tampering with the brain’s EM field directly should have zero latency in itself, or a separate signature. If the latency signature of neural alteration is the only kind ever found then EMF consciousness should ultimately need to be dismissed. But if in certain situations there were strong evidence of exact temporal correlation then electromagnetic consciousness might become the more simple explanation. Also consider how appropriate it would be for standard testing like EEG to help assess these matters. Unlike TMS, the point of this sort of testing would be to cause field effects that are as similar as possible to the field effects associated with standard neuron firing. With sensitive enough instruments researchers should be able to directly “see” the influences of their exogenous transmissions.
My own understanding is that EMF consciousness could be quite local in a perception sense, and this is because what’s felt needn’t affect the brain but rather just the field that would reside as consciousness. So it could be that extremely tiny but highly complex local energies from around the brain create all that’s seen, heard, tasted, and so on, and even if they affect nothing except the field itself. Theoretically a thinker would in some form also exist within the field that accepts such input for assessing. It boggles my mind to wonder if a thinker would require its own dedicated neural firing for it to exist, or rather a thinker would be inherent to the created desires. Regardless this thinker would interpret phenomenal input in the attempt to grasp what to do to make itself feel better from moment to moment. Beyond sensing outside world input it would automatically be provided with memories on the basis of what it was thinking about. So just as other senses are provided, I expect that memory resides as an EMF sense input provided to the thinker from neural firing from around the brain. Furthermore algorithms should lead the brain to provide the EMF thinker with appropriate memories, so once again the thinker may not need to affect the brain in general for such perceptions to exist. The only element of the brain which this EMF should need to affect would be motor neurons that take its instructions regarding muscle function. Furthermore I suspect that the brain sets up most of this automatically. Here the serial thinker essentially sets off “a chain of dominos” already created by the brain’s massively parallel function. So anyway, that’s where the field should need to be strong enough to affect the brain for ephaptic coupling. If motor neurons tend to reside in one essential location then this field needn’t be quite as strong in general. If the motor neurons it needs to affect are scattered in various brain locations however, as in the case of memories residing in various brain locations, then that would suggest a need for higher energies so that the deciding thinker could potentially affect them so that those instruments could be used.
On Orch-OR, McFadden actually wanted to promote this in the final chapter of his first published quantum biology book in 2000. After reading the Penrose and Hammeroff book on the matter however he realized it was a dead end. This was fortunate however since it led him to complete the final chapter of his quantum biology book with a classical EMF consciousness proposal. Chronologically that put him about as early as Pocket, though I don’t think he’s ever considered her true competition since her proposal is admittedly epiphenomenal.
On functionalism positing multiple realizability in terms of computer substrate, I think this should be considered uncontroversial. I’m a functionalist in that sense too. The actual magic of functionalism resides in how it mandates that there be no consciousness substrate. This specifically is what makes it unfalsifiable rather testing whether different styles of computers can do similar sorts of things. Is it possible to empirically explore “something” as consciousness? Of course. The converse however is inherently unfalsifiable. That’s why they suspect consciousness will arise in our machines before we even know it and so we should worry about exactly when they should be given human rights. And why our computers ought to become smarter than us and so take over our world. And why they hope human consciousness will ultimately be uploaded to technological machines so that humanity can effectively shed its biology. Substrate based consciousness would eliminate some of their most precious beliefs.
Illusionists are famously bad at explaining what their position happens to be, but I think they like it this way so it’s more difficult to challenge them. While illusionists have never been able to teach me what they believe, Eric Schwitzgebel’s innocent/wonderful conception of consciousness, originally written to challenge Keith Frankish, made their position clear to me. Furthermore in a foolish bit of initial honesty, Frankish even admitted that he does believe in that sort of consciousness. The lesson I’ve taken is that illusionists merely deny any conception of consciousness that isn’t causal, though they also believe it can have no dedicated substrate from the presumption that this doesn’t violate their first premise. My point is that the second does violate the first. This I might be referred to as “a super illusionist”. This is to say that I don’t believe in the magic that they don’t believe in, and I also don’t believe in the magic that they do believe in.
I think it might help the thought experiment if we make it a bit simpler and more realistic, which we can do by making the cards and rule book all about arithmetic using unfamiliar symbols. Now we could even make this a real experiment.
Then the question becomes something like, does a calculator or an abacus plus appropriate rules really do maths or handle numbers, or are they just moving symbols around in a way that maps onto the correct operations? I think the latter makes more sense. A calculator or abacus cannot tell you what a number is or point to any examples of them like we can, so it cannot understand them in the same way that we do. Plus we might use the same symbols and operations on them to mean different things, eg we're doing addition in thousands and don't want to type 000 all the time.
Likewise a (non-multimodal) LLM doesn't mean the same thing by its words as we do, because it only has other words for its reference frame. So I think the Chinese room doesn't really understand Mandarin either.
What's missing, I think, is contextual understanding and reference. They do have some understanding, but not the full understanding, and they lack the understanding we take as primary eg numbers are how many of a thing there are. It's like they're at the casino and can play the games, but they can't cash in.
But I don't think it should be impossible to give it such understanding. We just need its symbols to be properly linked up with what they symbolise.
The calculator is a nice way to put it. As I said in my long comment below, I think understanding resides in the creator of the room or calculator. One cannot design a calculator without understanding maths.
The calculator brings up another thing I see as missing from Searle's room. There is no manipulation of symbols, no math possible there. It's strictly input ⇒ output. So, answering any math question requires an infinite index. In contrast, calculators are algorithmic, using, as you say, a set of symbol manipulation rules to do math.
I think using arithmetic misses something. With Chinese characters, there is no way for the man to figure out what the characters mean. But with numbers, he can figure out that 1+1=2 and 1+2=3 (even if the characters are initially unfamiliar). He can then figure out the rest of arithmetic and gain an understanding. This is not possible with the Chinese characters.
Yes, I think you are all onto something. Mathematics might be special because we can (at least in many cases) work entirely within a formal system. The symbols don’t need to point to things out in the world—they just need to follow rules of manipulation that are internally consistent.
Of course, in practice, we humans often tie math back to meanings—like counting objects or measuring real distances. Joseph, I think your example is clever: it forces us to ask whether any manipulation of math symbols—absent of reference to real-world quantities—counts as “real” understanding.
It seems that you all might be coming from slightly different angles, but it also seems like everyone’s circling the same intuition: symbol manipulation doesn’t quite get us to understanding.
Meaning already has embedded in it a notion of mind. It has to do with a feeling of congruence between the external world and my internal representation of it. So, it already is qualia of a sort. Conceivably with sufficient access to the external world, the man in the "Room" would eventually come to understand Chinese in the same way I might learn it by living in Beijing and having access to a translation app on my phone.
Your comment raises a really interesting question: what exactly is it about that sufficient access to the world that makes understanding possible? Is it simply having sensory data (visuals, sounds, etc.)? Or is it about getting feedback? Or is there something else at play— like the need for a direct access? or felt experience?
It is about learning in my view and generating neural patterns from input. Possibly one of the main functions of consciousness is matching inputs with internal patterns. As we learn, the patterns become more precise and individualized to the exact input. When the congruence of pattern and input is sufficient (as judged by consciousness?), we have understanding. Of course, the understanding could still be wrong if the inputs become distorted or filtered (Mr. McGoo).
In the original Room, there is no learning because there is no feedback. There are no consequences for correct or incorrect responses. Just like in the McGoo cartoon. In the thought experiment, we assume the person carries out the instructions carefully so correct responses are being generated.
Magoo is from the US 50's but the cartoons are instructive in reminding us that things can seem one way, even consistently so, but be quite another way in fact.
On the thought of the Chinese room experiment, I'm inclined to agree with Searle. A human can have some idea or thought, represent that thought with an organized series of strokes on paper. Then another human can look at that group of strokes on paper and formulate roughly the same thought that the first human started with. Same goes for the series of sounds that we make called "speech." I've always been fascinated by our ability to do this, but a computer can't really do that the way we can, no matter how much it seems like it.
Computers can mathematically represent the writing we give it as input, then invoke certain subroutines which result in an output in the form of writing that us humans can recognize. The computer doesn't have the same understanding of the complex ideas that the writing represents in the way we do. Even the most advanced and capable AI in the world today has not deviated from this reality. It's important to keep this in mind since LLMs have given us a very convincing illusion otherwise.
Even Myelin, the neural networking engine that I'm developing myself, which is already capable of neuroplasticity, isn't the same thing as the actual brain. It's merely an analogy to it. It won't have the same understanding of ideas that humans have. It won't have sentience, nor the ability to experience.
I would agree that symbol manipulation doesn't equate to understanding nor meaning; it is merely a symptom thereof.
The operator in the room is an analogue for the CPU and of course the CPU has no understanding of the bits it manipulates. The understanding comes from The Architect who created the room. I don't think we can view Searle's room without considering its design.
Which I think takes it beyond the usual systems response. Arguably, even the whole room doesn't understand Mandarin. It is The Architect who designed the system that necessarily does. So, Searle's response doesn't work — it's still just the room. Nor does it work for the robot response. The understanding comes solely from The Architect. (Likewise, any understanding from an LLM comes from the data it has absorbed, not the data processing that spits out an answer.)
Even the wrong model response ignores the *design* of the rulebook. Compare the gym crowd blindly following an unknown syntax with the semantics we humans acquire as we learn about the world. We know what the various symbols mean or learn what they mean in context (or through education). The gym crowd by definition isn't learning anything.
As an aside: when I wrote about this long, long ago on a blog far, far away, I called Searle's room the GFR — the Giant File Room. One issue with his GFR is that, as presented, there is no mechanism for it to learn new material. A bigger one is that it has no algorithmic processes for computing new responses. A GFR would have to be truly infinite to answer all possible simple math questions. There are an infinite number of math questions for which the answer is "42", which requires an infinite index of input symbols to output the Mandarin symbols for "42".
I think your final question is indeed the truly interesting one. I suspect it is the acquisition of information and its contexts that provides meaning. The training of LLMs is a crude form of this and is perhaps why they've leaped so far ahead of the symbolic Ai of old. Searle's GFR and those old symbolic systems are what used to be called "expert systems", and their limitations became apparent early on.
Certainly the person in the room could build up the rules and data for generating responses as long as he/shet had the appropriate range of inputs and its outputs received feedback from the external world. The person would still not understand Chinese and there wouldn't be an Architect that understood Chinese either. That's essentially what our LLMs are doing.
I didn't say they did. I said they understand information and how to design a system for processing it. The "understanding" of the LLM comes from the information it digested.
And we're more and more discovering the limits of LLMs. For instance, asked a math question they often deliver a correct response (because of the training data) but when asked how they derived that answer, their reply doesn't at all match the actual process used.
There is another defense of the 'system reply' against Searle's "commit the book to memory" attempt to refute it: humans can learn to perform all sorts of complex procedures without understanding them. As a simple example, children learn how to do arithmetic without, at least initially, understanding the semantics of carrying, borrowing and shifting, and quite a few live out their lives without ever being able to explain it correctly (it is something of a problem even for many elementary-school teachers, apparently!) Furthermore, gaining proficiency in mental arithmetic neither requires nor grants this understanding.
Other examples include encryption-decryption and celestial navigation.
Attempts to show that it is different in the case of Searle's example face a significant empirical challenge from recent advances in technology, as, while machine translation and LLMs are far from perfect, it would now seem tendentious to insist that Searle's instruction book could not possibly be written (whether it could be executed by a person, either with pencil and paper or in their head, is beside the point.) This presents quite a dichotomy: either the task requires intelligence, or it does not. If it does, then Searle needs a better response to the system reply, but if it does not, then the Chinese Room argument simply fails to address the feasibility of strong AI.
The same can be said for Searle's response to the 'wrong model' reply.
Can I be cheeky and say... it depends on the social skills of the room!! If the room has okay social skills, it might just say: “Great.” If it’s a bit more polished: “I’m feeling great today, thanks for asking!” And if it really excels: “I’m feeling great today — how about you?!”
It’s funny (and a bit eerie) how easy it is to make the Room look socially competent! 😉
I suppose then it would work equally well to have it respond, “None of your business!” 😆
Which brings up another point. As defined, the Room would always give the same answer to the same prompt. In contrast, LLMs deliberately randomize their outputs — not always picking the words with the highest probability — exactly so they don’t come off as so obviously mechanistic.
Is it too obvious to say that the reply would simply be the result that came from the set of instructions the person followed? The person wouldn't understand the question or the response.
That seems a given because of how the Room is defined. Input ⇒ Output. But it seems a giveaway to the Room being purely mechanistic, because one would expect the answer to change over time.
Just a quick thought on 'understanding' and 'meaning' - what if the message passed in says something like "A fire is headed this way, evacuate." While the rulebook might come up with a response like "Thanks for the warning" if there's no resulting action then it seems like there's a lack of understanding of the meaning (or import) of the message. Are there simple tests that can prove understanding?
Or how about a more purely language-based test - send in message: "Please answer the next five messages with the same reply each time - 'I don't understand your message'". Does the box have memory?
By an interesting coincidence, in my philosophy class today, we were discussing whether justified-true-belief counts as knowledge if you don't have understanding.
Let's say that an expert knows a fact (eg, there are six kinds of quarks) and she tells you the fact. You now have testimony from an expert about a true fact that you now believe, but you don't understand what a quark is. Does that count as knowledge?
We decided that understanding is necessary before you can call it knowledge. Testimony is not enough. LLMs don't have understanding, so they can't have knowledge.
I like the Robot Reply above, which says that an AI's experience in the world can give it understanding. An AI can know that Paris is the capital of France, but without the robot body, it doesn't understand what 'Paris' or 'France' are. With the robot body, the AI can associate meaning with the words and understand.
Perhaps ChatGPT could gain true understanding if we put it in the back of a Tesla.
For anyone interested, I wrote a short article on how modifying the Chinese Room into the Searlese Room (emulating Searle's body & brain) makes Searle refute himself, titled "Searle vs Searle" https://markslight.substack.com/p/the-philosopher-in-the-machine
-----
Great article, as always, Suzi!
I didn't know about Fodors criticism! Very reasonable of him. It's so sloppy to just point at stuff and say "that's not X" (understanding, intentionality, qualia, a representation consciousness) without even attempting to specify what X means. Like just dismissing Dennett’s "Quining Qualia" on the grounds that "that's not what qualia is!".
What counts as symbol manipulation, in Searle’s mind? In your mind? In the commonsense? (I sincerely don't quite understand what is meant by symbol manipulation). Any program programmed in a programming language using symbols (all of them, I presume)? Binary code manipulation?
If so, connectionist AI supervenes on symbol manipulation. It supervenes on discrete on/off logic gates. Moreover, all the physical causal structures in human bodies can (presumably) be represented and calculated by symbol manipulation = supervene on symbol manipulation. So if Searle thinks human understanding is a thing that physical human bodies do, he better think that symbol manipulation can do it too.
Notably, LLMs are programmed (not trained) as symbol manipulation, and then they are trained to be connectionist and the connectionist AI can operate on symbols. And the AI can (hypothetically) program a physics simulation precise enough to evolve biological life and human-like intelligences.
It's also worth noting that an LLM does not inherently "know" that it is operating on symbols when taking in tokens and spitting out tokens. And it's also worth noting that whenever one speaks with someone else over the phone or internet, all of the meaning and intonation and timing is conveyed in discrete symbols. And that individual neuronal action potentials are discrete events (perhaps there are nuances here that you know and I don't - I seem to recall it can matter how "strong" a deploarisation is).
I'm not arguing against you here Suzi; as I read you, you largely agree or at least sympathise with much of my rants . The "Chinese Room" is just such a trigger for me and it takes out the worst in me. The fact that it's it's been taken so seriously makes me sad. But the fact that it has undeniably means we gotta deal with it. It's also what makes philosophy of mind interesting - if everything were intuitive and everyone agreed that would perhaps be boring.
You're dealing with this very nicely and clearly!
I think the title of your essay should be used for self-inquiry for anyone taken in by the Chinese Room argument. Imagine you were to peek into your own skull with the right tools to inspect all the individual parts of your brain. Then ask "Hello? Is there anybody in there?"
You’re right: the question of what counts as symbol manipulation gets hand-wavey fast, which usually means we’re circling something we don’t fully understand yet. At its simplest, a symbol is just something that stands in for something else. For example, the word dog is a symbol that represents the furry animal; a stop sign is a symbol that tells us to halt; and a mathematical variable like x can be seen as a symbol that represents a number.
Symbol manipulation, then, just means applying rules to these symbols — moving them around, transforming them, or recombining them. In certain cases, we can do this without necessarily understanding what the symbols mean. Classic examples include a calculator adding numbers, or a computer program transforming strings of code based on predefined logic.
There also seems to be a broad and a narrow sense of “symbol.” Vectors in LLMs, for instance, aren’t symbols in the traditional sense (like words or icons that directly stand for something). Vectors are mathematical representations — high-dimensional points that encode statistical patterns. So while LLMs process inputs and outputs that look like symbols (tokens of language), what’s happening under the hood is less like symbol manipulation in the classic sense and more like pattern-matching in a continuous space.
And yes! The Chinese Room sparks strong feelings because it feels so intuitively right to many people. But I think that’s exactly what makes these discussions worth having. I think there is value in slowing down and untangling these ideas carefully.
This puzzle is solved just by letting go of our need to compare AI to humans. It seems more useful to think of AI as a new species with it's own properties.
What you’re saying Lee, makes me wonder — if there’s a system within the brain that turns “brain noise” into “brain information”, then couldn’t the brain thus directly be classified as “an information processor”? And regardless of that question, I wonder what brain system you’re proposing gets this done?
I can see what you’re saying about “lynch-pin of structure” in an epistemic sense Lee. I get the concept. Mind you that in an ontological sense I consider all of causality tied together as a single system which requires each part for the structure to be whole, so one can’t be more important than another. In the end it should all be the same unified thing, or causality itself. But given this qualification, sure, there are things that seem more fundamental to us.
Regarding a brain lynch-pin itself, I think it’s pretty well established that the synaptically gated firing of neurons resides as a processing structure for input information. Here the brain may be considered a massively parallel computer. Essentially input information (such as from a thumb whack) gets algorithmically processed. Then the processed information goes on to do various things, such as release hormones and operate muscles. There should be nothing semantic here yet and so it should essentially be like one of our computers operating a robot. This perspective suggests that there must be a loop in brain function which creates the value dynamic that we feel, such as when a thumb gets damaged. So I presume the processed information informs something causally appropriate to exist, for example, as the experiencer of a thumb whack. The only thing that I know of that might serve this purpose is the electromagnetic field associated with the correct sort of neural firing. But I wonder if you’re going to propose a second reasonable possibility? Or instead are you going to challenge my position that the synaptically gated firing of neurons may usefully be said to reside as non-semantic information processing?
I understand that computational and functional perspectives aren’t everyone’s cup of tea. My newsletter, especially the recent series on information — and this new one on meaning — lean on those frameworks for two reasons:
1. I am a computational neuroscientist, so I am naturally going to draw on my areas of expertise. Computationalism is one side. Biology is the other half of my training. These two sides don't always fit neatly together. The tension between them can spark useful questions.
2. Computational functionalism is one of today’s most influential views, so I think it deserves a fair look — at both its strengths and its weaknesses.
That said, all viewpoints are welcome here. Like you, many readers do not subscribe to functionalism. And I’m happy (within time constraints) to try to understand their views. As I see it, diverse perspectives only enrich the conversation.
Would it help if we separate “information” from “meaning”? If we use information in the Shannon sense, then we might agree that the brain processes loads of information (in the Shannon sense). The question then turns to — how that information becomes meaningful.
Hi! You raise a genuinely important issue about where meaning comes from.
There is a lot of confusion around this term “information.”
In everyday language, information already means something (semantics is baked in). But in information theory (and in fields like neuroscience), information is simply structured variation that reduces uncertainty. It’s about patterns and correlations, even before anyone interprets them.
Imagine a temperature sensor hooked up to a thermostat. The sensor spits out a voltage proportional to the outside temperature. By itself, that voltage isn’t “about” anything in a human, semantic sense—but it still carries Shannon information, because tighter voltages map onto narrower ranges of possible temperatures. The thermostat can act on that pattern with zero need to “understand” what ° Celsius means.
This double use of the word information creates a lot of confusion.
Especially, when scientists say the brain is an information-processing system. When neuroscientists call the brain an information-processing system, they mostly mean it transforms, compresses, and routes those raw, pattern-laden signals so an organism can perceive, learn, and act. And you're absolutely right that as explained, this transformation alone doesn’t necessarily explain meaning.
You put your finger on the key question right at the end: what kind of system is needed to construct meaningful information from raw input? That’s exactly where I hope to take this series in the coming weeks. Looking forward to reading your thoughts.
Pardon my interjection, but I've noticed that for some reason First Cause can be breathtakingly rude and arrogant. I'm not sure he's aware how offputting it can be, but it's time someone mentioned it. In hopes of adding some levity, I can only recommend Fry and Laurie's "American Ass sketch." (https://www.youtube.com/watch?v=N8qw3oWvDpo)
I don't know what might have transpired in these threads since I last visited, but I'm sorry if I've offended you.
I'm sorry you feel that way, Lee. I saw that you had written some earlier comments, but it looks like you’ve since deleted them. I was planning to respond -- you had raised some interesting points.
A very good intro to the Chinese Room argument, and the common criticisms against it. I think this thought experiment is the epitome of what's wrong with taking philosophical thought experiments as authoritative in the same manner that actual experiments are. Searle's argument is rhetoric for a certain set of intuitions.
On the Wrong Model reply, it's worth pointing out that any artificial neural network you've used to date was implemented with a symbolic program. Neuromorphic computing might change that, but right now, ANNs are built on a symbolic foundation. IIT advocates use this fact to argue that they're still the wrong kind of causal structure.
But to Searle's reply, none of his neurons understand English. In fact, without Broca's area and Wernicke's area, neither does his brain. Yet with all the regions working together, we say he does understand it, which is the System's Reply used for his brain.
But the biggest issue I've always had with this thought experiment is the amount of time it would take Searle to reply to any question. Responding to even the simplest question would likely involve following billions, if not hundreds of billions of instructions. If we assume Searle can perform one a second, and doesn't take sleep, meal, or bathroom breaks, he might finish his first billion instructions in about 30 years. We can make him more productive by wheeling in a computer to help him, but now the original intuition of the thought experiment is weakened, or it should be.
To me, the best thing that can be said about the Chinese Room is it spurs discussion. But what it really does is clarify people's intuitions.
A computation taking one second on a machine or two million years in the "Room" are equivalent as long as the same steps are executed in the same order, aren't they?
They are. Although it's worth noting that relative to the environment, the one second version, assuming it has the right outputs, is much easier to intuit as comprising understanding.
"Easier to intuit" but still the same result. Isn't that what the thought experiment is about?
The whole point of the thought experiment is to drive the intuition that understanding can't possibly be there. But millenia or eons of manual activity compressed into seconds? Maybe that doesn't change the intuition for some, but it does for me. (Not that I think trusting intuitions in this area is productive anyway.)
Thanks, Mike!
Yes, the Chinese Room, like many other thought experiments, is “rhetoric for a certain set of intuitions.” Many famous thought experiments — like Mary’s Room or the Trolley Problem — also trade on intuition.
I think this hits on something interesting: what are thought experiments that trade on intuition good for, if anything? If they help highlight our intuitions, that could actually be a strength, right? They bring those intuitions out into the open. From there, we can ask: do our intuitions hold up when we test them against empirical evidence?
I like the phrase Dennett uses — intuition pump. The Chinese Room thought experiment gets us to feel the intuition that understanding must be more than symbol manipulation. But to test whether that intuition is correct, we’d need to bring in logic and/or empirical work. Maybe that’s where the mistake happens — assuming intuition tells us something true.
I guess that’s what you meant by: “the best thing that can be said about the Chinese Room is it spurs discussion. But what it really does is clarify people's intuitions.” That feels spot on.
Good point about the Wrong Model reply, too. LLMs are trained on symbolic text and often operate on discrete tokens at the first layer, so they’re not exactly symbol-free. If we think a system that computes over symbols can’t understand, wouldn’t a system that computes over vectors and matrices face the same problem? Neuromorphic computing is an interesting twist. Though, I could imagine someone arguing that if it’s built the same way our current models are built, it would still run into the same problem.
Thanks Suzi!
Right. I don't want to give the impression that I think thought experiments are useless. But we should remember in which way they're useful, not as arbiters of what reality is, but in flushing out our intuitions. In my view, the best ones show the limits of those intuitions, where they they break down or become contradictory.
But I'll admit to having a particular attitude toward Searle's. Maybe it's Searle's acerbic delivery, or the way I frequently encounter people who only seem to take account of his side, and not the many responses.
But most likely it's just the reliance on intuition as a guide to reality. It seems like the history of science shows that intuitions can be useful starting points for an investigation, but not as anything authoritative.
Haha, yes—the famous John Searle snark! I remembered he was frank, but re-reading his classic papers for this essay, I was reminded again how sharp-edged his tone really is. He peppers his argument with swipes at other views, and after a while, I started to wonder what’s driving that. It’s funny how sometimes the more confidently someone presses their point, the less confident we become in their claims.
Outstanding article. I understood the Chinese room argument conceptually before, but never really broke it down like this, and this brings it into question in my mind. Thank you for your writing.
Thanks so much, Fred!
Nicely done! Succinct, comprehensive and short introduction to a fascinating debate I was around to witness as a young man. I’m looking forward to this series and glad you’re back. All the best, John.
Thanks, John! It’ll be great to get your take on whether the debate has really moved beyond the original talking points — or if we’re still hashing out the same old puzzles.
Awesome Suzi! It’s probably well known here already that I agree with Searle. In 2019 however I felt like I needed to develop a thought experiment that in some ways goes further than his Chinese room. Instead of the vague concept of “understanding” that functionalists can easily claim is possessed by a hypothetical future computer that passes the Turing test for several months, I use the distinct concept of “thumb pain” that we should all grasp similarly. Furthermore I identify the problem of computational functionalism to be one of failing to complete the “output” component required for any input (like a key stroke), to effectively do anything. Should an experiencer of thumb pain exist when paper with the correct marks on it gets algorithmically processed to create more paper with the correct other marks on it? Of course not! To be causal the output paper should need to inform something causally appropriate that would itself reside as such an experiencer. Not only does this go beyond Searle’s case in that regard, but I even float the possibility that processed brain information goes on to inform a neurally produced electromagnetic field as the general substrate of consciousness. Then Google’s Notebook LM program took the text of my post and created an incredible podcast that breaks down my argument into more comprehensible plain speech units. How could this AI have enhanced what I’d written so much with just a couple minutes of processing, when if humanly produced it would have cost big money? Today they give away algorithms for free that are good enough to trick us into believing we’re dealing with people!
https://eborg760.substack.com/p/post-3-the-magic-of-computational
Ah, good—yes! There are some interesting arguments in the literature suggesting we should distinguish between the link between understanding and language, and the link between understanding and other experiences like pain, moods, and bodily responses. That might be an important distinction. But I'll get to that in an upcoming essay.
For now, I want to turn back to an earlier conversation we were having. I want to make sure I fully understand your view, because I’m still not getting something, and I'm not sure what I'm missing.
If I’m following: where functionalism says consciousness could in principle be implemented in any substrate that has the right causal structure, you’re saying—no, the substrate itself matters critically. Is that right?
And in your view, that critical substrate is EM fields. You’re saying EM fields are consciousness. So when you talk about information needing to “inform something causally appropriate,” the causally appropriate thing is the EM field. Yes?
Here’s where I get a bit stuck. In other comments, you’ve suggested that we could test your idea by generating weak EM fields that mimic those typical of neural firing, without those fields altering the firing of any neurons. But in that case, wouldn’t the EM field have no causal power? It’s not doing anything to change the system’s behaviour, right?
So I’m curious: how does your view avoid running into the same kind of problem that epiphenomenalism faces—where you’ve got a conscious field that exists but doesn’t cause anything?
Great questions Suzi! The last one is the big one so hopefully we can straighten this out. I’ll take them in order.
“…where functionalism says consciousness could in principle be implemented in any substrate that has the right causal structure…”
I’d alter this a bit because as I understand it computational functionalists don’t believe there can be “a consciousness substrate”, and especially when they also call themselves “illusionists”. What you might have meant to say there was something more like “consciousness could in principle be implemented by means of any computer with sufficient processing capacity”? And actually we’re in agreement there. I’d say the difference between us is that perceptions of consciousness is what they focus on while I go deeper to a potential causal substrate. Thus they might posit that thumb pain will reside by means of processed information in itself (and so no substrate that’s informed by it), while I posit that processed information can only exist as such to the extent that it informs something causally appropriate (or a potential consciousness substrate).
“And in your view, that critical substrate is EM fields.”
I’d say something more nuanced like “While I can see how processed brain information might be informing a neurally produced electromagnetic field to exist as consciousness, from my own grasp of brain function I’m not aware of a second reasonable possibility.”
“In other comments, you’ve suggested that we could test your idea by generating weak EM fields that mimic those typical of neural firing, without those fields altering the firing of any neurons. But in that case, wouldn’t the EM field have no causal power? It’s not doing anything to change the system’s behaviour, right?”
That’s the true question that I think’s been keeping you, and maybe various others, from grasping my position. So if we can now sort this out then I should be able to better clarify the matter in my next post.
Let’s say we induce exogenous field energies in someone’s brain that thus alters their neuron firing, as in the case of Transcranial Magnetic Stimulation. If this also alters someone’s consciousness then we shouldn’t know if it was the altered firing that created this effect in an unknown way, or rather the altered field itself was the consciousness that the exogenous energy interfered with. And regardless, because brains don’t produce energies anywhere near TMS strength anyway, I don’t know how one might effectively argue that consciousness exists by means of those sorts of energies that aren’t found by means of endogenous brain function.
Let’s now consider the opposite approach. We implant transmitters in various interesting parts of the brain that produce energies which ought to constructively and destructively alter the EM field associated with typical synchronous neuron firing, though shouldn’t be strong enough to affect brain function directly. Furthermore let’s say that a test subject’s experience of vision resides in the form of EMF associated with the correct synchronous neural firing. If (1) a transmitter close enough to the neuron firing that creates the field that’s experienced as vision (2) were to produce an energy that constructively and/or destructively alters the vision EMF, then (3) vision ought to become distorted in various ways. Here the EMF experiencer who knows they’re suppose to report such disturbances, might indeed correspond. From this theory the field/experiencer ephaptically couples with motor neurons that send signals to speech muscles for such report. Therefore that’s my plan — hopefully specialists would say that the exogenous energies shouldn’t be strong enough in themselves to directly affect neural function, but if EMF consciousness itself does become affected then apparently this larger energy structure (the thinker, theoretically) would go on to affect neural function for standard oral report. If these exogenous energies were reliably reported to distort someone’s consciousness, or could eventually even be tailored to give someone a specific type of experience, then what would a second explanation for this be?
Eric, I’ll reply to you here as well, as you referenced Suzi and your response to her. Suzi is pointing to the problem of epiphenomenalism here, and I think you’re not appropriately adressing it! Let me attempt to use your own kind of argument to show this.
You’ve been clear in our previous exchanges that the EMF theory is not a “strong emergence” theory, and that it respects the laws of physics as we know them on Earth. You posit the EMF as the 'causally appropriate thing' that processed information informs.
So, let's take that premise to its logical conclusion. Imagine we use 'marks on paper,' as you say, to perfectly compute Earth's entire physical evolution from a past state, say at some point in 2018, to this very moment in 2025. This grand calculation, strictly obeying all known physical laws, would encompass every atom, every force, and crucially, all electromagnetic fields in their entirety, including those generated by brains like yours. This comprehensive physical simulation would, therefore, chart the precise sequence of events leading to you becoming a proponent of EMF theories, your development of the thumb pain and marks-on-paper examples, every employment of that example, your writing of this entire post. It would also account for every interaction you and I have had, as well as my own journey to computational functionalism.
If these calculations—these 'marks on paper' representing the complete, causally closed physical system of Earth—show how the physical EMF (generated by neural activity and interacting according to Maxwell's equations and quantum mechanics) fully accounts for the neural states that lead to you thinking, writing, and arguing that this EMF is the seat of conscious experience, then what specific, additional causal work is being done by a 'non-computational' or 'non-functional' property of this EMF? If all the effects of the EMF, including your arguments about it, are already predicted and explained by its computable physical interactions within this simulation, then any proposed 'experience' or 'causally appropriate informed status' beyond these physically determined effects becomes indistinguishable from an epiphenomenon—a consequence, perhaps, but not an independent cause of your arguments or any other physical event.
Within the constraints of physical laws, whatever they may be, the ‘marks on paper’ argument undermines and ultimately refutes itself.
To tidy up the confusion around functionalism. Functionalism holds that any physical system that instantiates the right causal (in practice, computational) organisation would realise the same conscious states. That’s substrate-independence, not substrate-absence: some physical stuff must still be doing the work.
Illusionists then argue that what needs explaining is why those physical/computational processes lead us to judge that we have ineffable feels. They don’t deny the neural substrate; they deny that the ‘feels’ are anything over and above it.
Now, onto the interesting ideas...
To make sure I’m understanding you correctly, I’ll break it down as I see it—you can let me know where I’ve misunderstood you.
1. I think we’re on the same page about raw field strength
A figure-8 TMS pulses top out around 1.5 – 2 T at the coil face and are strong enough to depolarise cortical axons, eliciting phosphenes or motor twitches depending on the target site. Endogenous brain fields live many orders of magnitude lower—roughly 5 – 100 nT for magnetism and < 5 mV mm⁻¹ for electric gradients.
So yes, there’s a huge power gap.
2. But as I understand it, field-strength isn’t what EM-consciousness theories lean on
McFadden, Pockett and others locate conscious content in the spatiotemporal patterns of those weak endogenous fields, not in brute amplitude. A giant, homogeneous TMS pulse seems like the wrong pattern.
3. Can we really not tell whether TMS changes consciousness via neurons or “the field itself”?
I think we can. There have been a number of papers that show this. The experiments can be a bit tricky to read, but essentially, the experiments record both brain activity and reports and they show that tACS or patterned TMS first entrain neuronal firing; the subjective effects (e.g., flickering phosphenes, phase-dependent brightness shifts) track the phase of that entrainment, not the raw field amplitude. Somatosensory input can be blocked and the entrainment persists, pointing to a direct neuron–field interaction rather than a free-floating field phenomenology.
Here's a 2020 paper published in PLOS Biology that shows this idea: https://journals.plos.org/plosbiology/article?id=10.1371%2Fjournal.pbio.3000834
4. Why the “silent-field” transmitter is a physics long-shot
Any external source powerful enough to reshuffle the brain’s endogenous pattern must, by Maxwell’s laws, induce charge movement and nudge membrane potentials. Slice and modelling work shows that < 1 mV mm⁻¹—the very regime needed to re-paint the pattern—already shifts spike timing and network synchrony.
In other words: if you can sculpt the field, you’ve already touched the neurons.
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006974
https://www.sciencedirect.com/science/article/abs/pii/S0959438814001809?via%3Dihub
5. What alternative explains a reported distortion?
Plain-vanilla neural entrainment seems to cover it: the transmitter perturbs local spikes (directly or ephaptically), those spikes ripple through the circuit, and the subject reports a change. I don't see a need to reify the field as the experiencer.
So, the way I see it:
Field theories remain interesting — weak endogenous EM patterns may well modulate, coordinate or even help bind neural activity. But current evidence seems to suggest that the field does its work through the neurons that create it. If a future experiment can imprint a complex conscious scene by reshaping the field while leaving spikes untouched... well, that would indeed be revolutionary. Until then, Occam's razor seems like the best approach. And because “neurons first, fields as influence” remains the simpler — and better-tested — story, it seems like the most reasonable working hypothesis to me.
Thanks for so diligently considering my testing proposal Suzi! This should help improve my next essay. I’ll start with your “4. Why the “silent-field” transmitter is a physics long-shot”.
To help me assess this question I went back through McFadden’s first published paper on the matter, dated January 2002. (https://philpapers.org/archive/MCFSFA.pdf). Indeed, the suggestion there was also that EM fields always have at least some potential effects on neurons that are close enough to the source. It even mentioned that extremely small field effects from the firing of just one neuron can have some potential field effects on approximately 200 neighboring neurons. Instead I was under the impression that neurons are somewhat protected from at least minor field effects given their myelin sheathing — apparently that just helps create faster ionic transfers. I guess it’s mainly the computers we build that require shielding from EMF energies. Still the sensitivity of neural function to EM influences does at least suggest a greater potential role for the ephaptic coupling that EMF consciousness would require for information to loop from the field back to the brain. The net effect as I understand it is that the field effects of implanted transmitters should cause certain neurons that are on the edge of firing anyway, to be more likely or even less likely to do so.
Anyway I was wrong to hope that my proposed test would be reasonably clean in this sense. Thus if subjects were to report strange consciousness effects by means of appropriate exogenous transmissions from within their heads, then scientists would still need to determine if it’s because of field effects or rather neural effects? It’s hard to quite know how they might get around this issue until actual experiments are done. Would they learn to use exogenous EM energies to give people novel phenomenal experiences in the form of distinct images or sounds? In that case it would seem more likely that they were directly altering a subject’s consciousness rather than indirectly doing so by means of neural firing alterations that create such effects.
Furthermore observe that there’s another way such testing might go. When exogenous energies are generated that ought to alter the field associated with ongoing synchronous neural firing, it could be that subjects would not notice or report the occurrence of strange phenomenal circumstances. With enough such results, McFadden’s theory would then need to be dismissed. I’m not currently aware of any truly falsified consciousness proposals to date. I presume this is because beyond his, consciousness theories seem unfalsifiable in general. Do you know of any consciousness theories that have been empirically disproven, or like McFadden’s, could potentially be?
Actually beyond your point 3 that it ought to be possible to understand if TMS alters consciousness by means of neural effects or direct EMF effects, McFadden’s paper also suggests such a possibility. At one point he mentioned TMS which incites neural function that doesn’t exactly stop when the exogenous energies stop, though rather milliseconds later. Here it was presumed that exogenous transmissions can cause chains of neural firing that thus continue a bit longer. This tendency might help scientists with my proposed test as well. If phenomenal effects from transmitters in the head are reported to end exactly when transmissions end, and perhaps corroborated with monitoring like EEG, then this would be a sign that consciousness was altered directly by means of exogenous field effects. So that would help his case. But if phenomenal effects from transmitters in the head were found to continue on for milliseconds later, then this would suggest the incitement of endogenous neural firing. That should at least not help his case.
On functionalism mandating computational substrate independence, yes I agree. And indeed, I’m a functionalist in that sense too — I don’t think it should matter what substrate does the information processing. We differ regarding what comes next however. I believe that information should only be said to exist as such to the extent that it goes on to inform something causally appropriate. Thus pending evidence, processed brain information might be found to inform a neurally produced electromagnetic field which itself exists as what sees, hears, thinks, and so on. Functionalists however do not entertain substrate based consciousness proposals as I understand it, thus mandating an inherently unfalsifiable position. Furthermore it should be difficult for them to voluntarily postulate that consciousness might required a causal substrate because then they’d lose various long cherished sci-fi implications, such as a coming need for robots to be given human rights, a coming “singularity” where they become far more intelligent than us and so take over our world, and that humans might someday shed their biology for their consciousness to reside by means of technological machine function.
I consider illusionists essentially the same, but see them more as the attack faction in the sense that they like to tell people about the supernatural components to their various conceptions of consciousness. We’re aligned on that, though I go one step further to observe the spookiness of their substrate-absence based conception of consciousness. Yes the computer is considered to have substrate, though the consciousness it creates is proposed to exist as nothing in itself.
No worries, Eric! Like you, I love these sorts of conversations. 😊
On your point about myelin — yes, myelin speeds up the transmission of electrical signals. It does this by preventing the loss of electrical current across the membrane, allowing for faster conduction along long axons.
But in relation to your ideas: while myelin insulates axons, the bulk of the electric fields that form local field potentials actually arise in unmyelinated dendrites and somata in the cortex. So myelin doesn’t really “shield” the structures most involved in generating or sensing endogenous fields.
The standard view on EM interactions seems to have shifted over the years. Ephaptic effects — where electric fields influence nearby neurons — are now fairly uncontroversial and are getting more attention in research. But most measured effects are still weak (<1 mV) and highly local.
That said, ephaptic coupling isn’t always that tiny. While classic cortical studies reported <0.5 mV shifts, recent work on the cerebellum has shown multi-millivolt fields strong enough to pause the firing of dozens of nearby Purkinje cells up to ~60 µm away. These are powerful micro-scale synchronisers — but still a far cry from the kind of brain-wide, content-rich field a McFadden-style theory would require.
It’s super interesting work, though. There’s a lot to explore in this space!
Here are a few links you might find helpful:
A recent review on ephaptic coupling:
https://www.sciencedirect.com/science/article/pii/S0301008223000667?via%3Dihub
The “Spooky Action Potentials at a Distance” piece from Harvard (original paper is linked at the bottom):
https://brain.harvard.edu/hbi_news/spooky-action-potentials-at-a-distance-ephaptic-coupling
A recent follow-up to that work:
https://elifesciences.org/articles/88335
As for invasive stimulation devices (like depth electrodes delivering oscillatory electric fields) — yes, they clearly modulate perception. But so far, the data suggests that these effects originate from synaptic-network activity, not some “silent-field” route. No one has demonstrated a clean dissociation. Everything seems tightly coupled, and the causal arrows all point to neural activity.
You also mentioned that TMS after-effects persist for milliseconds after the pulse. Definitely true. A single TMS pulse can evoke bursts of cortical firing beginning 1–4 ms after stimulation, and those reverberations can last tens to hundreds of milliseconds. Behavioural and EEG effects tend to track that neural activity.
These studies are hard to run! I had a rough time trying to record simultaneous TMS or tDCS with EEG. The very same pulses or currents you use to stimulate the brain swamp the EEG electrodes with artefact. So we had to spend a lot of time designing hardware and analysis pipelines just to recover the actual brain signals underneath the noise.
For more on that front, here's a link to a Frontiers Research Topic from 2017—11 papers on the causal role of neural oscillations in perception and cognition:
https://www.frontiersin.org/research-topics/4919/rhythmic-noninvasive-brain-stimulation-nibs-investigating-the-causal-role-of-neural-oscillations-for-perceptual-and-cognitive-processes/magazine
On whether consciousness theories can be falsified, I think you raise an important point. Many are, unfortunately, not. But I’d argue that the Orch-OR microtubule model is an exception — and that it has been seriously undermined. The calculated decoherence times in the brain are far shorter than the theory requires, making its central mechanism physically unworkable.
As for functionalism, some would say it is falsifiable. It posits multiple realisability, but makes testable predictions: if two systems share the same causal organisation, they should exhibit the same cognitive or behavioural profile. It could be challenged by finding a system where causal structure is preserved but cognition/behaviour clearly differs. So I’d say functionalism remains empirically open — not automatically unfalsifiable, though of course, finding a decisive test is notoriously tricky.
And finally, illusionism. I think a lot of people misread what illusionists are actually claiming. It can sound like they’re saying we don’t have experiences at all — but that’s not it. Thinkers like Dennett and Frankish argue that we do have experiences — it's just that our introspection leads us to believe they are something they are not.
Some of your points seem particularly hopeful to me Suzi! So specialists are already implanting depth brain electrodes that deliver oscillatory electric fields? For my proposed test to even potentially be attempted, I’ve been worried that unique new brain transmitters would need to be invented. Of course the purpose of these current instruments must be to alter neural firing given that few today suspect consciousness itself might exist under the brain’s EMF. Hopefully such instruments could be calibrated to simulate the synchronous firing of specific numbers of neurons, maybe in the tens, hundreds, or thousands? Ideally this could even be varied on the fly just as individual neuron firing can vary on the fly. Mind you that a successful test wouldn’t necessarily have to be “silent” to neural effects. Directly affecting neurons would simply add the uncertainty of whether a given documented consciousness alteration was neural or field sourced. And your point about myelin EMF protection could be helpful as well since a given depth transmitter might be implanted in an area where neurons are more heavily myelinated and thus a bit more protected from a transmitter’s influences.
There’s something else that you said which might ultimately make such testing quite feasible. If there tends to be a 1 – 4 ms phenomenal alteration delay after stimulation, with effects lasting for tens to hundreds of ms after a given TMS pulse ends, then shouldn’t this serve as a signature of neural tampering? Conversely notice that the light speed function of tampering with the brain’s EM field directly should have zero latency in itself, or a separate signature. If the latency signature of neural alteration is the only kind ever found then EMF consciousness should ultimately need to be dismissed. But if in certain situations there were strong evidence of exact temporal correlation then electromagnetic consciousness might become the more simple explanation. Also consider how appropriate it would be for standard testing like EEG to help assess these matters. Unlike TMS, the point of this sort of testing would be to cause field effects that are as similar as possible to the field effects associated with standard neuron firing. With sensitive enough instruments researchers should be able to directly “see” the influences of their exogenous transmissions.
My own understanding is that EMF consciousness could be quite local in a perception sense, and this is because what’s felt needn’t affect the brain but rather just the field that would reside as consciousness. So it could be that extremely tiny but highly complex local energies from around the brain create all that’s seen, heard, tasted, and so on, and even if they affect nothing except the field itself. Theoretically a thinker would in some form also exist within the field that accepts such input for assessing. It boggles my mind to wonder if a thinker would require its own dedicated neural firing for it to exist, or rather a thinker would be inherent to the created desires. Regardless this thinker would interpret phenomenal input in the attempt to grasp what to do to make itself feel better from moment to moment. Beyond sensing outside world input it would automatically be provided with memories on the basis of what it was thinking about. So just as other senses are provided, I expect that memory resides as an EMF sense input provided to the thinker from neural firing from around the brain. Furthermore algorithms should lead the brain to provide the EMF thinker with appropriate memories, so once again the thinker may not need to affect the brain in general for such perceptions to exist. The only element of the brain which this EMF should need to affect would be motor neurons that take its instructions regarding muscle function. Furthermore I suspect that the brain sets up most of this automatically. Here the serial thinker essentially sets off “a chain of dominos” already created by the brain’s massively parallel function. So anyway, that’s where the field should need to be strong enough to affect the brain for ephaptic coupling. If motor neurons tend to reside in one essential location then this field needn’t be quite as strong in general. If the motor neurons it needs to affect are scattered in various brain locations however, as in the case of memories residing in various brain locations, then that would suggest a need for higher energies so that the deciding thinker could potentially affect them so that those instruments could be used.
On Orch-OR, McFadden actually wanted to promote this in the final chapter of his first published quantum biology book in 2000. After reading the Penrose and Hammeroff book on the matter however he realized it was a dead end. This was fortunate however since it led him to complete the final chapter of his quantum biology book with a classical EMF consciousness proposal. Chronologically that put him about as early as Pocket, though I don’t think he’s ever considered her true competition since her proposal is admittedly epiphenomenal.
On functionalism positing multiple realizability in terms of computer substrate, I think this should be considered uncontroversial. I’m a functionalist in that sense too. The actual magic of functionalism resides in how it mandates that there be no consciousness substrate. This specifically is what makes it unfalsifiable rather testing whether different styles of computers can do similar sorts of things. Is it possible to empirically explore “something” as consciousness? Of course. The converse however is inherently unfalsifiable. That’s why they suspect consciousness will arise in our machines before we even know it and so we should worry about exactly when they should be given human rights. And why our computers ought to become smarter than us and so take over our world. And why they hope human consciousness will ultimately be uploaded to technological machines so that humanity can effectively shed its biology. Substrate based consciousness would eliminate some of their most precious beliefs.
Illusionists are famously bad at explaining what their position happens to be, but I think they like it this way so it’s more difficult to challenge them. While illusionists have never been able to teach me what they believe, Eric Schwitzgebel’s innocent/wonderful conception of consciousness, originally written to challenge Keith Frankish, made their position clear to me. Furthermore in a foolish bit of initial honesty, Frankish even admitted that he does believe in that sort of consciousness. The lesson I’ve taken is that illusionists merely deny any conception of consciousness that isn’t causal, though they also believe it can have no dedicated substrate from the presumption that this doesn’t violate their first premise. My point is that the second does violate the first. This I might be referred to as “a super illusionist”. This is to say that I don’t believe in the magic that they don’t believe in, and I also don’t believe in the magic that they do believe in.
I think it might help the thought experiment if we make it a bit simpler and more realistic, which we can do by making the cards and rule book all about arithmetic using unfamiliar symbols. Now we could even make this a real experiment.
Then the question becomes something like, does a calculator or an abacus plus appropriate rules really do maths or handle numbers, or are they just moving symbols around in a way that maps onto the correct operations? I think the latter makes more sense. A calculator or abacus cannot tell you what a number is or point to any examples of them like we can, so it cannot understand them in the same way that we do. Plus we might use the same symbols and operations on them to mean different things, eg we're doing addition in thousands and don't want to type 000 all the time.
Likewise a (non-multimodal) LLM doesn't mean the same thing by its words as we do, because it only has other words for its reference frame. So I think the Chinese room doesn't really understand Mandarin either.
What's missing, I think, is contextual understanding and reference. They do have some understanding, but not the full understanding, and they lack the understanding we take as primary eg numbers are how many of a thing there are. It's like they're at the casino and can play the games, but they can't cash in.
But I don't think it should be impossible to give it such understanding. We just need its symbols to be properly linked up with what they symbolise.
The calculator is a nice way to put it. As I said in my long comment below, I think understanding resides in the creator of the room or calculator. One cannot design a calculator without understanding maths.
The calculator brings up another thing I see as missing from Searle's room. There is no manipulation of symbols, no math possible there. It's strictly input ⇒ output. So, answering any math question requires an infinite index. In contrast, calculators are algorithmic, using, as you say, a set of symbol manipulation rules to do math.
I think using arithmetic misses something. With Chinese characters, there is no way for the man to figure out what the characters mean. But with numbers, he can figure out that 1+1=2 and 1+2=3 (even if the characters are initially unfamiliar). He can then figure out the rest of arithmetic and gain an understanding. This is not possible with the Chinese characters.
I’m loving this whole thread — such great points!
Yes, I think you are all onto something. Mathematics might be special because we can (at least in many cases) work entirely within a formal system. The symbols don’t need to point to things out in the world—they just need to follow rules of manipulation that are internally consistent.
Of course, in practice, we humans often tie math back to meanings—like counting objects or measuring real distances. Joseph, I think your example is clever: it forces us to ask whether any manipulation of math symbols—absent of reference to real-world quantities—counts as “real” understanding.
It seems that you all might be coming from slightly different angles, but it also seems like everyone’s circling the same intuition: symbol manipulation doesn’t quite get us to understanding.
Meaning already has embedded in it a notion of mind. It has to do with a feeling of congruence between the external world and my internal representation of it. So, it already is qualia of a sort. Conceivably with sufficient access to the external world, the man in the "Room" would eventually come to understand Chinese in the same way I might learn it by living in Beijing and having access to a translation app on my phone.
This is great, thanks James!
Your comment raises a really interesting question: what exactly is it about that sufficient access to the world that makes understanding possible? Is it simply having sensory data (visuals, sounds, etc.)? Or is it about getting feedback? Or is there something else at play— like the need for a direct access? or felt experience?
It is about learning in my view and generating neural patterns from input. Possibly one of the main functions of consciousness is matching inputs with internal patterns. As we learn, the patterns become more precise and individualized to the exact input. When the congruence of pattern and input is sufficient (as judged by consciousness?), we have understanding. Of course, the understanding could still be wrong if the inputs become distorted or filtered (Mr. McGoo).
https://en.wikipedia.org/wiki/When_Magoo_Flew
In the original Room, there is no learning because there is no feedback. There are no consequences for correct or incorrect responses. Just like in the McGoo cartoon. In the thought experiment, we assume the person carries out the instructions carefully so correct responses are being generated.
Yes, that feels feels intuitively right to me too. I had never heard of Mr. Magoo before -- but yes, what a great analogy!
Magoo is from the US 50's but the cartoons are instructive in reminding us that things can seem one way, even consistently so, but be quite another way in fact.
Something we should be reminded of often!
I like that definition of meaning:
Meaning = a feeling of congruence between the external world and my internal congruence of it.
On the thought of the Chinese room experiment, I'm inclined to agree with Searle. A human can have some idea or thought, represent that thought with an organized series of strokes on paper. Then another human can look at that group of strokes on paper and formulate roughly the same thought that the first human started with. Same goes for the series of sounds that we make called "speech." I've always been fascinated by our ability to do this, but a computer can't really do that the way we can, no matter how much it seems like it.
Computers can mathematically represent the writing we give it as input, then invoke certain subroutines which result in an output in the form of writing that us humans can recognize. The computer doesn't have the same understanding of the complex ideas that the writing represents in the way we do. Even the most advanced and capable AI in the world today has not deviated from this reality. It's important to keep this in mind since LLMs have given us a very convincing illusion otherwise.
Even Myelin, the neural networking engine that I'm developing myself, which is already capable of neuroplasticity, isn't the same thing as the actual brain. It's merely an analogy to it. It won't have the same understanding of ideas that humans have. It won't have sentience, nor the ability to experience.
I would agree that symbol manipulation doesn't equate to understanding nor meaning; it is merely a symptom thereof.
Another excellent essay, Suzi.
The operator in the room is an analogue for the CPU and of course the CPU has no understanding of the bits it manipulates. The understanding comes from The Architect who created the room. I don't think we can view Searle's room without considering its design.
Which I think takes it beyond the usual systems response. Arguably, even the whole room doesn't understand Mandarin. It is The Architect who designed the system that necessarily does. So, Searle's response doesn't work — it's still just the room. Nor does it work for the robot response. The understanding comes solely from The Architect. (Likewise, any understanding from an LLM comes from the data it has absorbed, not the data processing that spits out an answer.)
Even the wrong model response ignores the *design* of the rulebook. Compare the gym crowd blindly following an unknown syntax with the semantics we humans acquire as we learn about the world. We know what the various symbols mean or learn what they mean in context (or through education). The gym crowd by definition isn't learning anything.
As an aside: when I wrote about this long, long ago on a blog far, far away, I called Searle's room the GFR — the Giant File Room. One issue with his GFR is that, as presented, there is no mechanism for it to learn new material. A bigger one is that it has no algorithmic processes for computing new responses. A GFR would have to be truly infinite to answer all possible simple math questions. There are an infinite number of math questions for which the answer is "42", which requires an infinite index of input symbols to output the Mandarin symbols for "42".
I think your final question is indeed the truly interesting one. I suspect it is the acquisition of information and its contexts that provides meaning. The training of LLMs is a crude form of this and is perhaps why they've leaped so far ahead of the symbolic Ai of old. Searle's GFR and those old symbolic systems are what used to be called "expert systems", and their limitations became apparent early on.
Certainly the person in the room could build up the rules and data for generating responses as long as he/shet had the appropriate range of inputs and its outputs received feedback from the external world. The person would still not understand Chinese and there wouldn't be an Architect that understood Chinese either. That's essentially what our LLMs are doing.
Even LLMs have architects who understand information and who designed the process of digesting and processing it effectively.
Those architects do not need to understand Chinese either.
I didn't say they did. I said they understand information and how to design a system for processing it. The "understanding" of the LLM comes from the information it digested.
And we're more and more discovering the limits of LLMs. For instance, asked a math question they often deliver a correct response (because of the training data) but when asked how they derived that answer, their reply doesn't at all match the actual process used.
Didn't you write "the understanding comes from The Architect?"
In the case of Searle's room (or calculators), yes, of course.
There is another defense of the 'system reply' against Searle's "commit the book to memory" attempt to refute it: humans can learn to perform all sorts of complex procedures without understanding them. As a simple example, children learn how to do arithmetic without, at least initially, understanding the semantics of carrying, borrowing and shifting, and quite a few live out their lives without ever being able to explain it correctly (it is something of a problem even for many elementary-school teachers, apparently!) Furthermore, gaining proficiency in mental arithmetic neither requires nor grants this understanding.
Other examples include encryption-decryption and celestial navigation.
Attempts to show that it is different in the case of Searle's example face a significant empirical challenge from recent advances in technology, as, while machine translation and LLMs are far from perfect, it would now seem tendentious to insist that Searle's instruction book could not possibly be written (whether it could be executed by a person, either with pencil and paper or in their head, is beside the point.) This presents quite a dichotomy: either the task requires intelligence, or it does not. If it does, then Searle needs a better response to the system reply, but if it does not, then the Chinese Room argument simply fails to address the feasibility of strong AI.
The same can be said for Searle's response to the 'wrong model' reply.
Searle's defences of this argument are incredibly simplistic. But, did we expect otherwise?
Beautiful article, one of the clearest exposition that I have read of the Chinese's room debate.
Thank you so much, Juan José!
One question I've had is what kind of reply the Room would make to the question, "How are you feeling today?"
Haha, great question!
Can I be cheeky and say... it depends on the social skills of the room!! If the room has okay social skills, it might just say: “Great.” If it’s a bit more polished: “I’m feeling great today, thanks for asking!” And if it really excels: “I’m feeling great today — how about you?!”
It’s funny (and a bit eerie) how easy it is to make the Room look socially competent! 😉
I suppose then it would work equally well to have it respond, “None of your business!” 😆
Which brings up another point. As defined, the Room would always give the same answer to the same prompt. In contrast, LLMs deliberately randomize their outputs — not always picking the words with the highest probability — exactly so they don’t come off as so obviously mechanistic.
Is it too obvious to say that the reply would simply be the result that came from the set of instructions the person followed? The person wouldn't understand the question or the response.
That seems a given because of how the Room is defined. Input ⇒ Output. But it seems a giveaway to the Room being purely mechanistic, because one would expect the answer to change over time.
Just a quick thought on 'understanding' and 'meaning' - what if the message passed in says something like "A fire is headed this way, evacuate." While the rulebook might come up with a response like "Thanks for the warning" if there's no resulting action then it seems like there's a lack of understanding of the meaning (or import) of the message. Are there simple tests that can prove understanding?
Or how about a more purely language-based test - send in message: "Please answer the next five messages with the same reply each time - 'I don't understand your message'". Does the box have memory?
hahaha! I love it
Wonderful as always, Suzi!
By an interesting coincidence, in my philosophy class today, we were discussing whether justified-true-belief counts as knowledge if you don't have understanding.
Let's say that an expert knows a fact (eg, there are six kinds of quarks) and she tells you the fact. You now have testimony from an expert about a true fact that you now believe, but you don't understand what a quark is. Does that count as knowledge?
We decided that understanding is necessary before you can call it knowledge. Testimony is not enough. LLMs don't have understanding, so they can't have knowledge.
I like the Robot Reply above, which says that an AI's experience in the world can give it understanding. An AI can know that Paris is the capital of France, but without the robot body, it doesn't understand what 'Paris' or 'France' are. With the robot body, the AI can associate meaning with the words and understand.
Perhaps ChatGPT could gain true understanding if we put it in the back of a Tesla.
Love this — what a perfect coincidence!
Your comment actually got me thinking about something... I’ll come back to this once I’ve sorted out my thoughts.
For anyone interested, I wrote a short article on how modifying the Chinese Room into the Searlese Room (emulating Searle's body & brain) makes Searle refute himself, titled "Searle vs Searle" https://markslight.substack.com/p/the-philosopher-in-the-machine
-----
Great article, as always, Suzi!
I didn't know about Fodors criticism! Very reasonable of him. It's so sloppy to just point at stuff and say "that's not X" (understanding, intentionality, qualia, a representation consciousness) without even attempting to specify what X means. Like just dismissing Dennett’s "Quining Qualia" on the grounds that "that's not what qualia is!".
What counts as symbol manipulation, in Searle’s mind? In your mind? In the commonsense? (I sincerely don't quite understand what is meant by symbol manipulation). Any program programmed in a programming language using symbols (all of them, I presume)? Binary code manipulation?
If so, connectionist AI supervenes on symbol manipulation. It supervenes on discrete on/off logic gates. Moreover, all the physical causal structures in human bodies can (presumably) be represented and calculated by symbol manipulation = supervene on symbol manipulation. So if Searle thinks human understanding is a thing that physical human bodies do, he better think that symbol manipulation can do it too.
Notably, LLMs are programmed (not trained) as symbol manipulation, and then they are trained to be connectionist and the connectionist AI can operate on symbols. And the AI can (hypothetically) program a physics simulation precise enough to evolve biological life and human-like intelligences.
It's also worth noting that an LLM does not inherently "know" that it is operating on symbols when taking in tokens and spitting out tokens. And it's also worth noting that whenever one speaks with someone else over the phone or internet, all of the meaning and intonation and timing is conveyed in discrete symbols. And that individual neuronal action potentials are discrete events (perhaps there are nuances here that you know and I don't - I seem to recall it can matter how "strong" a deploarisation is).
I'm not arguing against you here Suzi; as I read you, you largely agree or at least sympathise with much of my rants . The "Chinese Room" is just such a trigger for me and it takes out the worst in me. The fact that it's it's been taken so seriously makes me sad. But the fact that it has undeniably means we gotta deal with it. It's also what makes philosophy of mind interesting - if everything were intuitive and everyone agreed that would perhaps be boring.
You're dealing with this very nicely and clearly!
I think the title of your essay should be used for self-inquiry for anyone taken in by the Chinese Room argument. Imagine you were to peek into your own skull with the right tools to inspect all the individual parts of your brain. Then ask "Hello? Is there anybody in there?"
Hey Mark! Thanks so much for the kind words!
You’re right: the question of what counts as symbol manipulation gets hand-wavey fast, which usually means we’re circling something we don’t fully understand yet. At its simplest, a symbol is just something that stands in for something else. For example, the word dog is a symbol that represents the furry animal; a stop sign is a symbol that tells us to halt; and a mathematical variable like x can be seen as a symbol that represents a number.
Symbol manipulation, then, just means applying rules to these symbols — moving them around, transforming them, or recombining them. In certain cases, we can do this without necessarily understanding what the symbols mean. Classic examples include a calculator adding numbers, or a computer program transforming strings of code based on predefined logic.
There also seems to be a broad and a narrow sense of “symbol.” Vectors in LLMs, for instance, aren’t symbols in the traditional sense (like words or icons that directly stand for something). Vectors are mathematical representations — high-dimensional points that encode statistical patterns. So while LLMs process inputs and outputs that look like symbols (tokens of language), what’s happening under the hood is less like symbol manipulation in the classic sense and more like pattern-matching in a continuous space.
And yes! The Chinese Room sparks strong feelings because it feels so intuitively right to many people. But I think that’s exactly what makes these discussions worth having. I think there is value in slowing down and untangling these ideas carefully.
This puzzle is solved just by letting go of our need to compare AI to humans. It seems more useful to think of AI as a new species with it's own properties.