Thanks, Jack! That's really interesting! It makes me wonder how much perception, memory and conscious experience are related. I suspect when players swear they can track the ball all the way into their glove, we are not simply talking about conscious experience. It brings up that fascinating question of how much of what we perceive in the moment is raw input, and how much is our brain's reconstruction based on memory, predictions, and assumptions?
I’m not much of a baseball fan, but I’ve heard of a phenomenon in baseball called 'losing the ball in the lights', where fielders drop an easy catch because the stadium lights are too bright — they simply don’t see the ball.
There’s some fascinating research suggesting that while glare from the lights does interfere with seeing the ball, something else might be happening, too. Even when players can still see the ball, the glare disrupts their ability to judge its trajectory and speed. Without accurate information about the ball’s motion, the brain struggles to predict where the ball will be. When the brain can’t make a reliable prediction, even experienced players miss catches they’d normally find easy.
Very nice finale to the series, Suzi! I'm curious to see what we end up learning from the investigation.
Do you think this is the reason why we have such a clear perception from our peripheral vision, even though the actual image itself is unfocused (to say the least). Or the same reason why, when we close our eyes, go blind, or otherwise lose all visual input, we can perceive auditory inputs much more clearly, and still have a relatively decent picture of what's going on in our environment? Like if we lose one way to predict our environment and the other entities within it, the brain rewires itself to make predictions based on other sensory input instead?
Great question! Yes, I think this is part of the story, but I also suspect there's some trickery at play. We often fall for the illusion that we "see" more than we actually do. There's a lot of evidence suggesting we don’t take in as much as we think. The popular (but controversial) idea that the brain "fills in" missing information is central here. Dennett is well-known for arguing that there’s no such "filling in," but recent evidence suggests it might occur — just not in the way we typically imagine.
What we do know for sure is that our peripheral vision is much worse than most people realise. Yet, we don’t experience the world as blurry or incomplete around the edges. According to predictive coding theory, the brain maintains a model of the world and primarily focuses on surprises—updating the model only when predictions are wrong. This is energy-efficient: the brain doesn’t need to process every detail, only the unexpected. So, it's not so much "filling in" but rather just "not updating."
This may explain why we’re so good at catching movement in the corner of our eye—movement is surprising and grabs the brain’s attention. And attention (we think) might be important for updating our model of the world.
You also make an excellent point about sensory compensation! When people lose their vision, their brain doesn’t simply give up on trying to predict and understand the environment — it adapts, relying more on other senses. As I mentioned in some articles I wrote last year, there’s fascinating research that shows how parts of the brain typically used for visual processing can be repurposed for sound and touch in blind individuals. The brain is incredibly flexible, but the predictive coding theory suggests one of the brain's primary jobs is to be a prediction machine — it strives to model the world and will do so by any means necessary.
Predictive coding theory... I haven't heard of it, but it sounds interesting; I'll do some research on it. Your description of it reminds me of how videos are rendered on a computer: only certain small regions of the screen are updated at a time, and is done for very similar efficiency gains. I'm not sure how accurate of a comparison that is though.
On the topic of sensory compensation, maybe I should check out some of those older articles. I do admire how flexible the brain is like that.
"This may explain why we’re so good at catching movement in the corner of our eye" or that part of the cornea is specialised to detect change and movement rather than producing a pretty photograph. It also has proportionally more rods so presumably is better after dark - the time when you most need all-round vision cos it's difficult to keep your predictions up to date in the dark.
As for filling in: this was established centuries ago when the blindspot was discovered. (Google says "Edme Mariotte (1620-1684) discovered the blind spot in the eye in the 1660s.") Why are people still arguing about it?
People are still arguing about 'filling in' because many see the idea of 'filling in' as a problem. If the brain 'fills in' the missing information, we might wonder, who the brain is doing the filling in for? Many believe that the idea of 'filling in' implies that there must be a centralised observer who views the finished 'filled in' picture. But there is no such observer.
The "centralised observer" is me when I notice the filling in has happened (or when I don't see the black spot).
Somebody must have looked at how the filling in works. Easy when it's just blue sky, more complicated when it's a scene. Obvious experiments with regular and irregular patterns... Maybe it infills in Fourier space.
Just realised I've been thinking static view of an image. If we move our eyes we can fill from memory.
Then again, do we actually fill in or do we just pay no attention to that bit so never notice the gap? (as Dennett and others have suggested)
Is there even a corresponding spot in the visual cortex with no signal from the retina? Apparently there is:
An excellent description of why the predictive processing theory has so much promise!
Another nice benefit is it's relatively easy to see how it could evolve. Predictions can start off extremely simple, little more than reacting to the precursors of a stimuli rather than waiting for the stimuli itself. But as distance senses develop, those predictions become increasingly further out.
Until we end up with a primate able to predict itself and the observable universe.
Yes, exactly! The evolutionary angle is fascinating, isn't it? You can imagine how even very simple organisms would benefit from being able to react to warning signs rather than waiting for the actual threat, like a simple creature that pulls back when it detects a change in light.
From there, the question is, is it simply a series of gradual improvements? As creatures develop better sensory systems, as we add more bits like memory, these systems are able to make predictions that reach further into the future. They are able to process more complex types of information, too. A mouse, for example, doesn't just react to changes in light -- it seems to have a model of its world -- it predicts where predators might be lurking and it knows safe routes for escape.
And then, as you point out, we reach the really mind-bending part: brains that are sophisticated enough to predict not just immediate threats or opportunities but abstract concepts, complex social situations, and even their own behaviour. Humans (and many other creatures) seem to run mental simulations of possible futures; we try to predict our world and what happens in it, but we also predict the intentions of others, and yes, this might mean we even predict ourselves.
So, the theory goes, our conscious experience is the latest development in a very long evolutionary story that started with the simplest of survival advantages.
This is essentially one interpretation of predictive coding theory — a theory that has been gaining a lot of traction lately. I agree it holds a lot of promise. There are some issues to work through, but that just makes it more interesting. How far can prediction explain consciousness?
"The present (somewhere between history and the future) arrives as fast as it escapes us. I gather therefore that it must be a future view of history that matters most, albeit with its decidedly and significantly limited forward visibility and its blurred perspective on things past."
"I’ve planned away in circumstances where manipulation, pretend control, was the only way to achieve what the world, God, would not give me. I’ve also let the imagination fly in turns of wished for futures and rewards, sometimes small, at times grandiose.
But it is true that not once, whether it was the imagination or the will that led the way, could I have represented in detail, in truth or with accuracy any of the results of my plans."
The claim that *all* our conscious experience is of the future (even just the very near future) would indeed be an extraordinary one - it would imply, for example, that if we are watching a light that will be turned on when a sensor detects a cosmic ray, we would experience the light switching on before it happens.
A very reasonable position is that our sensory perception lags behind the events causing it, but we have a limited ability to model what will happen next, and our conscious minds are either doing that modeling or are perceiving the results of an unconscious predictive process.
Is there anything puzzling about that? (Zeno might have said so!) Maybe there seems to be a problem in how a nervous system that cannot keep up with reality can get ahead in predicting it, but the prediction does not have to predict every instant - for example, I can easily predict that my bowling ball will fall into the gutter seconds before it does.
... "it would imply, for example, that if we are watching a light that will be turned on when a sensor detects a cosmic ray, we would experience the light switching on before it happens."
I often have the experience of clicking the mouse *after* the event it triggers.
I've experienced that sensation, too. Time in the brain is a really weird thing. Our experience of when things happen isn't as straightforward as we think.
Hey, Drew! Great comment! I didn't mean to suggest that our conscious experience is of the future — that we have built-in clairvoyance. That would indeed be an extraordinary claim!
What the research suggests is more nuanced. When we track predictable motion (like a ball flying through the air), our visual system appears to compensate for neural delays by predicting where the object will be. The key phrase here is predictable motion. The visual system only makes accurate predictions when the movement follows familiar patterns — like objects moving in straight lines at constant speeds.
Your bowling ball example is a great one. That's a case of prediction based on your knowledge and experience. Your visual system has built up an understanding of how objects typically move, and research suggests that when tracking such familiar motion patterns, your early visual cortex can respond in 'real-time', bypassing the usual sensory delay.
I'm definitely not suggesting we're psychic or can see the future! It’s more that our visual system has evolved clever strategies to work around its own processing delays. For predictable motion, it gets ahead of the curve by anticipating where things will be. But for unpredictable events (like your cosmic ray example), we're stuck with the usual limitations — in this situation, there's a delay between the event and the neurons processing it.
I agree that consciousness should essentially be the brain’s best guess of the now rather than just incoming information, since for many activities incoming information ought to be a bit late. And instead of merely the predictable game of catch, I like considering the battle in baseball between highly skilled pitchers and hitters. Big league hitting requires an incredible amount of instantaneous prediction skill, and learned over many years of daily practice. Though there should be conscious elements to hitting, for such learned activity we essentially hand things back to the non-conscious brain to take care of automatically. It’s the same for typing, driving, and most everything else that we often presume we’re doing “consciously”, though much of it is actually learned automatic brain function.
This is essentially my “dual computers” model of brain function. Here the brain is essentially a non-conscious energy driven computer that we can teach to do things for us, like hit baseballs and type, while consciousness is a product of the brain that’s essentially a value driven computer that thus seeks to feel as good as it can from moment to moment. In my first post I presented the instantaneous self to be somewhat joined with past selves by means of memory, and joined with potential future selves by means of hope and worry. I need to finally get a second post completed! With so many different ways to potentially go, I find it difficult to simply add another element and leave the rest for later.
Thanks, Eric! I’m really glad you brought up these points — they're excellent.
I think we can add even more complexity to the problem here. The brain processes information about objects (the what pathway) and information about location (the where pathway) through different pathways. The where pathway, which guides motor responses, is faster than the what pathway, which tells us 'it’s a ball'.
Now, if we imagine the brain as a purely bottom-up system for a moment (it's not, but let's pretend), our motor system might already be processing how to catch the ball before the brain fully registers what it's seeing. This means that even in a simple bottom-up model, our muscles might start moving before we've entirely processed what we're moving toward (or away from). Add feedback and prediction into the mix, and it gets very tricky to determine when things are happening in the brain.
As for structuring ideas in your newsletter — I can definitely relate! Figuring out what to say and when to say it is one of the trickiest parts of writing. For me, planning is both the most time-consuming and the most enjoyable part. There are so many angles to explore, and deciding what needs to be explained first can feel like solving a puzzle.
I’m not anywhere near as good as you are on the planning Suzi! And in truth your project is quite different from mine. You come at this from the perspective of a professional career. In these newsletters you not only help share your education, but also generally illuminate things that continue to confound science — the fun stuff! Conversely I decided long ago that certain basic flaws mandated the softness of fields like psychology, and so figured that such formal study (unlike economics) wouldn’t work for me. But that never stopped me from modeling such function anyway. My situation feels a bit to me like the math prodigy Matt Damon played in “Good Will Hunting”. His life was too messed up to make a go of academia, and yet he could do things that math professionals could not. The main difference is that math professionals aren’t uncertain about what good math happens to be. If the same were true for soft scientists however, then they’d instead be hard scientists. So when you post about perplexing mental and behavioral questions, I enjoy presenting sensible answers from a perspective that academia does not yet seem equipped to provide.
Regarding the complexities of separate “where” and “what” pathways, my dual computers perspective might help simplify things. Here I mean that 0% of the brain is conscious and so we shouldn’t speak in terms of it “seeing”, “tasting” and so on. Yes this massive computer has a sensitive camera, microphone, chemical analyzers, and other such input devices, but nothing phenomenal. Instead one of its outputs is to create a phenomenal non-brain computer that we call consciousness. (Here it should help to envision consciousness as an electromagnetic field that is both created by and can affect the brain, at least as a placeholder until experimentally confirmed or not.) So when the brain camera detects something moving fast towards the face, there should be algorithms set up to automatically cause evasion. Here consciousness becomes a spectator to such function. Furthermore some people train that “where” brain function to help them hit baseballs. But visually recognizing an object from that camera? First the brain must create the experiencer of such an image. Then for conscious recognition to take place the experienced image may need to be recycled through for a while so that an appropriate memory leads to a sensible recognition. So yes, that could be quite slow. Of course to speed things up we often just assume what things are given their context.
Prediction is only one function of consciousness, it is not all there is. I fell into this trap after reading "How Emotions Are Made" by Lisa Barrett Ph.D. Then I realized, cognition and contemplation are not predictions. Also the experience of higher consciousness is non-sensory and non-predictive. We also have the subconscious mind running our lives (and helping us catch balls) and the preconscious, which Freud posited and is often overlooked, but an important human experience based on my own observations. Keep up the good work, Suzi!
Thanks, Buck! I’m really looking forward to getting into this topic with you. I think you make some strong points, but I’m not entirely convinced that cognition and contemplation are completely non-predictive. Take contemplation, for instance — aren’t we essentially 'running simulations' during the process? It feels like we’re making predictions about various scenarios or outcomes. While these predictions may be more abstract, they still seem to be predictions in a sense.
That said, you’ve touched on something really crucial about the unconscious mind. What’s fascinating is that it seems we can make (at least low-level perceptual) predictions even when we’re unconscious. This does suggest that prediction and conscious experience are not the same thing, which is such an intriguing idea to unpack. I'll get into that research and question later in the year. Looking forward to reading your thoughts on it.
Years ago I participated in an international discussion group on the topic of Complexity Science. Everyone passionately expressed their point of view, and no one agreed. This led me to the principle I call IADOYPOV - It All Depends On Your Point Of View. This is not relativism but an acknowledgement that your philosophical starting point determines your perception on any topic. As a former Psych major, I love the idea of constructionism which Barrett et al. advocate, but it has limits. There are also people who believe that consciousness occurs outside the body, or as one psychic put it, consciousness just sticks its toe into the body. Relatedly, "I think therefore I am" is not the same as "I think therefore I predict." As I sit here I am thinking about what to type next (predicting). But as I stop and look at my right hand, just look at it, I am not predicting. I am observing, perceiving. At least I think so. LOL!
That sounds like a fun discussion group to be involved in! I do like the 'It All Depends On Your Point Of View' idea -- I also think it depends on the questions you are asking.
I remember as a youth playing softball and catching at line drive hit directly at me. The only thing I remembered was the ball hitting my glove. Consciousness isn't all that involved in catching a ball once the skill is learned, it is heavily involved, however, in learning the skill in the first place.
Yes, I think you’ve hit on something really important here about the relationship between conscious learning and unconscious skill. I think it’s fascinating how, once we’ve mastered a skill, thinking too consciously about it can actually disrupt performance. But what I find particularly intriguing is why consciousness is even necessary during the learning phase. Why does the process of acquiring a skill seem to require this deliberate, conscious focus in the first place?
If consciousness is the mechanism for information integration (the mechanism NOT information itself as in IIT), then learning a skill would recruit information from many different parts of the brain. However, most of that information isn't required for the task. For example, you might start with 50 variables in a predictive model, but discover only 5 of them are needed to predict successfully.. Once the 5 are learned and refined, you can drop the other variables from the model. The high degree of information integration is no longer needed. The brain can do the task with less energy, always as important consideration evolutionarily since the brain demands for energy must be balanced with the body's demands.
The bigger the brain, the more refined the senses, the greater the requirements for integration across the brain.
When first starting out playing catch, you do get hit in the face. Over time, your entire nervous and musculature system shorten the reaction time -- and no doubt shorten the detection-reaction circuit pathway. Consider all the evidence in the natural world where reaction exceeds conscious consideration: a cat dodging a snake bite, a dragonfly shifting to snatch a mosquito, a falcon striking a pigeon from the air. Tens of milliseconds. I can't imagine much conscious "thinking" going on there.
Yes, exactly! Those lightning-fast reactions you describe — like a cat dodging a snake or a dragonfly catching its prey — happen incredibly quickly. The brain is fascinating in this regard. Visual information related to movement is processed in the dorsal "where" pathway, while information about objects is processed in the ventral "what" pathway. Interestingly, the "where" pathway (movement) is faster than the "what" pathway (object recognition). For creatures like cats, the "where" pathway is even faster than ours, giving them a significant edge in reaction speed.
If the brain operated as a purely bottom-up system — with information flowing in only one direction — we would always move before becoming aware of what we were reacting to. However, the brain is far from purely bottom-up. There are countless feedback projections that influence, or bias, the incoming information. This interplay between feedforward and feedback processes makes the question of when things happen in the brain incredibly complex and far from straightforward.
One thing that blows my mind still is that there is not one brain that makes predictions but thousands of cortical columns (think, CPU cores) that each process the stimuli in predictive fashion and then *vote* by comparing their results laterally. 🤯
Exactly! It is mind-boggling! We often talk about "the brain" as if it’s a single, unified processor, but as you’ve pointed out, it’s more like a massive parallel processing system, with different parts making their own “decisions” simultaneously. It’s almost as if the brain is made up of thousands of tiny machines, each working independently yet in coordination.
Thanks for highlighting this — it’s such a crucial point. And you’re absolutely right — we’re only just dipping our toes in!
Seems you’ve dug out (pun intended) the core of baseball—pitchers with curve balls, sliders, screw balls, etc. against batters who are gaining experience during each at bat. I’m not sure we always predict but sometimes wait to see what happens or try to figure out what happens before taking action (so I question the idea that this is the core to consciousness).
We probably don’t recall getting hit in the face as babies but all normally developing children smack themselves in the face and then cry about the question who did this to them? Eventually some control develops with the growing self preservation instinct/drive. Newborns don’t catch balls tossed at them.
I recall one instance when my wife and I were waiting at a signaled intersection to cross a street. We were standing there at the curb waiting with our first infant daughter in her stroller. Suddenly we heard car tires screeching to our left! Instantly and without thinking I grabbed the stroller handle and ran backwards away from the curb and street corner—my wife who had one hand on the stroller stood at the curb frozen. Then I saw a car sliding sideways heading toward us—her! I yelled—MOVE! The car screeched to a halt and the car behind it slammed into the back of it. The light had changed and the first driver didn’t want to run the light—apparently—and the car behind didn’t expect this sudden stop. When all the cars had stopped my wife turned and yelled at me for grabbing the stroller so quickly and I yelled back what was she doing standing at the curb with screeching cars heading towards US?—a different us that we were just getting used to. Her answer—she was trying to figure out what was happening before she decided what to do (a puzzling but perfectly reasonable response). My point —who cares what is happening, run and get away then figure out what was going on (more the attitude stick your hand in front of your face first then see if a ball hits it—and if so catch it!).
And clearly the two drivers were using different method of living in the moment or predicting the next moment. That is, until their multiple methods converged—as did their cars.
Magical performance ( regardless of whether it is slight if hand, street magic, prop magic, or illusion) has a similar theme to create the effect—set the expectations and tempt the audience to get ahead of you, then do the unexpected. Magic!
Oh wow, Dean! I’m so glad you, your wife, and your baby were okay—that’s quite the story. You’ve captured such an interesting dynamic here: the contrast between instinctive, predictive reactions (like you with the stroller) and the more deliberate, analytical approach (like your wife).
I like your point about magic -- magicians succeed precisely because they understand how our predictive brains work. They set up our expectations, let our brains make predictions about what will happen next, and then violate those predictions. In a way, they're hacking our brain's prediction systems!
Thank you for sharing your ideas, Dean! You have such a knack for making abstract ideas feel vivid and relatable and, as a consequence, you make us think.
"This has led to a radical alternative theory: conscious experience is not a simulation of what is happening now -- it's a prediction of what will happen in the (very near) future."
Perhaps because of my long career in predictive analytics, I don't view the prediction idea as particularly radical. Maybe conscious experience is a mix of both a perception of what is happening now together with a prediction of what will happen in the very near future. Based on their experiences, people (and animals) are constantly refining their models of how the world works, and they use those models to predict the future, from near to distant. I'm reminded of this whenever I watch a dog skillfully track and catch a flying frisbee.
As an example of a longer range prediction, I recall some experiment involving placing a cat in a room in front of an opaque screen, and having a person walk from left to right, passing temporarily behind one side of the screen and emerging from the other side back into the cat's visual field. At some age cats have learned enough about the continuity of moving objects (and creatures) to expect the person to reappear, and if he/she doesn't the cat acts surprised and may walk behind the screen to try to figure out where the person went.
Your example of playing catch with your nephew reminds me of a day, several decades ago, when I was playing catch with a member of my brother's family (perhaps a nephew). I was backing up to catch a ball he had thrown, but didn't notice the rock or tree stump just behind me, so I tripped and fell on my rear end. I hardly felt any pain at the time, but over the next few days my neck, which had apparently taken a jolt from the fall, started getting sore. The discomfort got worse and worse until I thought I would need to seek medical attention. But just at that point it started getting better and cleared up completely over the next few days. I speculated that I had suffered some kind of minor "whiplash" injury.
Predicting the future also happens while we are reading. Based on the words we have read so far we narrow down our expectations for the next word or words, and those expectations actually alter what we think we read. This is one reason people make typographical mistakes while writing: while reading back what they have written, they tend to "fill in the blanks" with their expectations and don't even notice a missing or incorrect word.
Thanks, Dave! You've got me wanting to watch cat and dog YouTube.
I’m with you — prediction doesn’t feel all that radical. I spent about 10 years in a lab studying how attention and prediction shape our experience. So, when you mention your long career in predictive analytics, I can’t help but think — that sounds fascinating!
I love all your examples, but the reading one really stood out to me. It highlights how prediction happens at multiple levels. For instance, we predict at the letter level (reading misspelled words as if they’re correct), the word level (filling in missing words), and even at the meaning level (where two people can interpret the same text differently, predicting different outcomes or meanings).
You might think the biggest danger in playing catch is getting hit in the face with the ball — but nope! It's that sneaky tree stump lurking behind you! I'm glad you're okay.
Your example "where two people can interpret the same text differently, predicting different outcomes or meanings" is relevant to the fields of law and diplomacy. Often, to get sufficient agreement to pass a law or ratify a treaty, its drafters must deliberately make the language sufficiently vague and ambiguous so the various stakeholders can interpret it to their satisfaction. But this is just "passing the buck". When later disputes arise about what the law or treaty actually means in practice, those ambiguities must somehow be addressed and resolved.
"Jimmy Carter knew how to get an audience to pay attention. In a speech given during the US President's 1977 visit to Poland, he appeared to express sexual desire for the then-Communist country. Or that's what his interpreter said, anyway. It turned out Carter had said he wanted to learn about the Polish people's 'desires for the future'."
The BBC article cited a few more mistakes by his interpreters during that visit, and concluded: "By the time Carter's trip ended, he had become the punchline for many a Polish joke."
That reminds me of Sofia Coppola's entertaining film "Lost in Translation" (2003), starring Bill Murray and Scarlett Johansson. Per Wikipedia, "The film explores themes of alienation and disconnection against a backdrop of cultural displacement in Japan.".
Good description. Predictive processing is a good model which I think is correct but incomplete. In my own view there is a need for a Marvel Universe version where the Hypotheses Assemble. My pattern matching model, predictive processing, Global workspace theory and maybe some IIT as topping.
Good point! I agree that focusing too much on one theory can be a mistake. Too often, we get stuck in our own frameworks and overlook alternative perspectives simply because they are not 'our theory'.
Perhaps consciousness is not like a camera? Some analogies lead to new discoveries, some don't. The analogy that consciousness is like a camera leads to perplexities. Why retain it?
This goes towards explaining why in professional baseball, a batting average of .300 (three hits out of ten tries) is considered very good. Fastballs in the 90+ MPH range give the batter only 400 milliseconds to react. And while it does seem some top-level players actually can see the ball, there is an element of guesswork involved for most batters. (This is exactly why sign-stealing can be productive for a cheating team. Gives the batter a clue of what's coming.)
I had to look up 'sign-stealing'. Yeah, that would help — especially when you only have 400 milliseconds to decide whether to swing or let it fly past. Sign-stealing puts a new spin (puns!) on what we mean by a calculated guess.
It gives one an appreciation for the level of skill at the major league level. Most of us wouldn’t even see the ball fly past.
Teams have come up with elaborate schemes, binoculars and a walkie talkie in the bleachers, and someone in the dugout making sounds the batter can hear, but sometimes, with a runner on second (who can see the catcher’s signs) it’s just a matter of something like adjusting the ball cap for a fastball or adjusting the belt for a curve or whatever. Anything for an edge!
Thanks, Jack! That's really interesting! It makes me wonder how much perception, memory and conscious experience are related. I suspect when players swear they can track the ball all the way into their glove, we are not simply talking about conscious experience. It brings up that fascinating question of how much of what we perceive in the moment is raw input, and how much is our brain's reconstruction based on memory, predictions, and assumptions?
I’m not much of a baseball fan, but I’ve heard of a phenomenon in baseball called 'losing the ball in the lights', where fielders drop an easy catch because the stadium lights are too bright — they simply don’t see the ball.
There’s some fascinating research suggesting that while glare from the lights does interfere with seeing the ball, something else might be happening, too. Even when players can still see the ball, the glare disrupts their ability to judge its trajectory and speed. Without accurate information about the ball’s motion, the brain struggles to predict where the ball will be. When the brain can’t make a reliable prediction, even experienced players miss catches they’d normally find easy.
haha! Sounds like your brain was wired for tennis!
Very nice finale to the series, Suzi! I'm curious to see what we end up learning from the investigation.
Do you think this is the reason why we have such a clear perception from our peripheral vision, even though the actual image itself is unfocused (to say the least). Or the same reason why, when we close our eyes, go blind, or otherwise lose all visual input, we can perceive auditory inputs much more clearly, and still have a relatively decent picture of what's going on in our environment? Like if we lose one way to predict our environment and the other entities within it, the brain rewires itself to make predictions based on other sensory input instead?
Great question! Yes, I think this is part of the story, but I also suspect there's some trickery at play. We often fall for the illusion that we "see" more than we actually do. There's a lot of evidence suggesting we don’t take in as much as we think. The popular (but controversial) idea that the brain "fills in" missing information is central here. Dennett is well-known for arguing that there’s no such "filling in," but recent evidence suggests it might occur — just not in the way we typically imagine.
What we do know for sure is that our peripheral vision is much worse than most people realise. Yet, we don’t experience the world as blurry or incomplete around the edges. According to predictive coding theory, the brain maintains a model of the world and primarily focuses on surprises—updating the model only when predictions are wrong. This is energy-efficient: the brain doesn’t need to process every detail, only the unexpected. So, it's not so much "filling in" but rather just "not updating."
This may explain why we’re so good at catching movement in the corner of our eye—movement is surprising and grabs the brain’s attention. And attention (we think) might be important for updating our model of the world.
You also make an excellent point about sensory compensation! When people lose their vision, their brain doesn’t simply give up on trying to predict and understand the environment — it adapts, relying more on other senses. As I mentioned in some articles I wrote last year, there’s fascinating research that shows how parts of the brain typically used for visual processing can be repurposed for sound and touch in blind individuals. The brain is incredibly flexible, but the predictive coding theory suggests one of the brain's primary jobs is to be a prediction machine — it strives to model the world and will do so by any means necessary.
Predictive coding theory... I haven't heard of it, but it sounds interesting; I'll do some research on it. Your description of it reminds me of how videos are rendered on a computer: only certain small regions of the screen are updated at a time, and is done for very similar efficiency gains. I'm not sure how accurate of a comparison that is though.
On the topic of sensory compensation, maybe I should check out some of those older articles. I do admire how flexible the brain is like that.
"This may explain why we’re so good at catching movement in the corner of our eye" or that part of the cornea is specialised to detect change and movement rather than producing a pretty photograph. It also has proportionally more rods so presumably is better after dark - the time when you most need all-round vision cos it's difficult to keep your predictions up to date in the dark.
As for filling in: this was established centuries ago when the blindspot was discovered. (Google says "Edme Mariotte (1620-1684) discovered the blind spot in the eye in the 1660s.") Why are people still arguing about it?
Good points!
People are still arguing about 'filling in' because many see the idea of 'filling in' as a problem. If the brain 'fills in' the missing information, we might wonder, who the brain is doing the filling in for? Many believe that the idea of 'filling in' implies that there must be a centralised observer who views the finished 'filled in' picture. But there is no such observer.
The "centralised observer" is me when I notice the filling in has happened (or when I don't see the black spot).
Somebody must have looked at how the filling in works. Easy when it's just blue sky, more complicated when it's a scene. Obvious experiments with regular and irregular patterns... Maybe it infills in Fourier space.
Just realised I've been thinking static view of an image. If we move our eyes we can fill from memory.
Then again, do we actually fill in or do we just pay no attention to that bit so never notice the gap? (as Dennett and others have suggested)
Is there even a corresponding spot in the visual cortex with no signal from the retina? Apparently there is:
https://journals.physiology.org/doi/full/10.1152/jn.01330.2004
An excellent description of why the predictive processing theory has so much promise!
Another nice benefit is it's relatively easy to see how it could evolve. Predictions can start off extremely simple, little more than reacting to the precursors of a stimuli rather than waiting for the stimuli itself. But as distance senses develop, those predictions become increasingly further out.
Until we end up with a primate able to predict itself and the observable universe.
Yes, exactly! The evolutionary angle is fascinating, isn't it? You can imagine how even very simple organisms would benefit from being able to react to warning signs rather than waiting for the actual threat, like a simple creature that pulls back when it detects a change in light.
From there, the question is, is it simply a series of gradual improvements? As creatures develop better sensory systems, as we add more bits like memory, these systems are able to make predictions that reach further into the future. They are able to process more complex types of information, too. A mouse, for example, doesn't just react to changes in light -- it seems to have a model of its world -- it predicts where predators might be lurking and it knows safe routes for escape.
And then, as you point out, we reach the really mind-bending part: brains that are sophisticated enough to predict not just immediate threats or opportunities but abstract concepts, complex social situations, and even their own behaviour. Humans (and many other creatures) seem to run mental simulations of possible futures; we try to predict our world and what happens in it, but we also predict the intentions of others, and yes, this might mean we even predict ourselves.
So, the theory goes, our conscious experience is the latest development in a very long evolutionary story that started with the simplest of survival advantages.
This is essentially one interpretation of predictive coding theory — a theory that has been gaining a lot of traction lately. I agree it holds a lot of promise. There are some issues to work through, but that just makes it more interesting. How far can prediction explain consciousness?
"
"The present (somewhere between history and the future) arrives as fast as it escapes us. I gather therefore that it must be a future view of history that matters most, albeit with its decidedly and significantly limited forward visibility and its blurred perspective on things past."
https://eme1998.substack.com/p/wherefrom-wisdom
And,
"I’ve planned away in circumstances where manipulation, pretend control, was the only way to achieve what the world, God, would not give me. I’ve also let the imagination fly in turns of wished for futures and rewards, sometimes small, at times grandiose.
But it is true that not once, whether it was the imagination or the will that led the way, could I have represented in detail, in truth or with accuracy any of the results of my plans."
https://eme1998.substack.com/p/planning
The claim that *all* our conscious experience is of the future (even just the very near future) would indeed be an extraordinary one - it would imply, for example, that if we are watching a light that will be turned on when a sensor detects a cosmic ray, we would experience the light switching on before it happens.
A very reasonable position is that our sensory perception lags behind the events causing it, but we have a limited ability to model what will happen next, and our conscious minds are either doing that modeling or are perceiving the results of an unconscious predictive process.
Is there anything puzzling about that? (Zeno might have said so!) Maybe there seems to be a problem in how a nervous system that cannot keep up with reality can get ahead in predicting it, but the prediction does not have to predict every instant - for example, I can easily predict that my bowling ball will fall into the gutter seconds before it does.
... "it would imply, for example, that if we are watching a light that will be turned on when a sensor detects a cosmic ray, we would experience the light switching on before it happens."
I often have the experience of clicking the mouse *after* the event it triggers.
I've experienced that sensation, too. Time in the brain is a really weird thing. Our experience of when things happen isn't as straightforward as we think.
Hey, Drew! Great comment! I didn't mean to suggest that our conscious experience is of the future — that we have built-in clairvoyance. That would indeed be an extraordinary claim!
What the research suggests is more nuanced. When we track predictable motion (like a ball flying through the air), our visual system appears to compensate for neural delays by predicting where the object will be. The key phrase here is predictable motion. The visual system only makes accurate predictions when the movement follows familiar patterns — like objects moving in straight lines at constant speeds.
Your bowling ball example is a great one. That's a case of prediction based on your knowledge and experience. Your visual system has built up an understanding of how objects typically move, and research suggests that when tracking such familiar motion patterns, your early visual cortex can respond in 'real-time', bypassing the usual sensory delay.
I'm definitely not suggesting we're psychic or can see the future! It’s more that our visual system has evolved clever strategies to work around its own processing delays. For predictable motion, it gets ahead of the curve by anticipating where things will be. But for unpredictable events (like your cosmic ray example), we're stuck with the usual limitations — in this situation, there's a delay between the event and the neurons processing it.
I agree that consciousness should essentially be the brain’s best guess of the now rather than just incoming information, since for many activities incoming information ought to be a bit late. And instead of merely the predictable game of catch, I like considering the battle in baseball between highly skilled pitchers and hitters. Big league hitting requires an incredible amount of instantaneous prediction skill, and learned over many years of daily practice. Though there should be conscious elements to hitting, for such learned activity we essentially hand things back to the non-conscious brain to take care of automatically. It’s the same for typing, driving, and most everything else that we often presume we’re doing “consciously”, though much of it is actually learned automatic brain function.
This is essentially my “dual computers” model of brain function. Here the brain is essentially a non-conscious energy driven computer that we can teach to do things for us, like hit baseballs and type, while consciousness is a product of the brain that’s essentially a value driven computer that thus seeks to feel as good as it can from moment to moment. In my first post I presented the instantaneous self to be somewhat joined with past selves by means of memory, and joined with potential future selves by means of hope and worry. I need to finally get a second post completed! With so many different ways to potentially go, I find it difficult to simply add another element and leave the rest for later.
Thanks, Eric! I’m really glad you brought up these points — they're excellent.
I think we can add even more complexity to the problem here. The brain processes information about objects (the what pathway) and information about location (the where pathway) through different pathways. The where pathway, which guides motor responses, is faster than the what pathway, which tells us 'it’s a ball'.
Now, if we imagine the brain as a purely bottom-up system for a moment (it's not, but let's pretend), our motor system might already be processing how to catch the ball before the brain fully registers what it's seeing. This means that even in a simple bottom-up model, our muscles might start moving before we've entirely processed what we're moving toward (or away from). Add feedback and prediction into the mix, and it gets very tricky to determine when things are happening in the brain.
As for structuring ideas in your newsletter — I can definitely relate! Figuring out what to say and when to say it is one of the trickiest parts of writing. For me, planning is both the most time-consuming and the most enjoyable part. There are so many angles to explore, and deciding what needs to be explained first can feel like solving a puzzle.
I’m not anywhere near as good as you are on the planning Suzi! And in truth your project is quite different from mine. You come at this from the perspective of a professional career. In these newsletters you not only help share your education, but also generally illuminate things that continue to confound science — the fun stuff! Conversely I decided long ago that certain basic flaws mandated the softness of fields like psychology, and so figured that such formal study (unlike economics) wouldn’t work for me. But that never stopped me from modeling such function anyway. My situation feels a bit to me like the math prodigy Matt Damon played in “Good Will Hunting”. His life was too messed up to make a go of academia, and yet he could do things that math professionals could not. The main difference is that math professionals aren’t uncertain about what good math happens to be. If the same were true for soft scientists however, then they’d instead be hard scientists. So when you post about perplexing mental and behavioral questions, I enjoy presenting sensible answers from a perspective that academia does not yet seem equipped to provide.
Regarding the complexities of separate “where” and “what” pathways, my dual computers perspective might help simplify things. Here I mean that 0% of the brain is conscious and so we shouldn’t speak in terms of it “seeing”, “tasting” and so on. Yes this massive computer has a sensitive camera, microphone, chemical analyzers, and other such input devices, but nothing phenomenal. Instead one of its outputs is to create a phenomenal non-brain computer that we call consciousness. (Here it should help to envision consciousness as an electromagnetic field that is both created by and can affect the brain, at least as a placeholder until experimentally confirmed or not.) So when the brain camera detects something moving fast towards the face, there should be algorithms set up to automatically cause evasion. Here consciousness becomes a spectator to such function. Furthermore some people train that “where” brain function to help them hit baseballs. But visually recognizing an object from that camera? First the brain must create the experiencer of such an image. Then for conscious recognition to take place the experienced image may need to be recycled through for a while so that an appropriate memory leads to a sensible recognition. So yes, that could be quite slow. Of course to speed things up we often just assume what things are given their context.
Prediction is only one function of consciousness, it is not all there is. I fell into this trap after reading "How Emotions Are Made" by Lisa Barrett Ph.D. Then I realized, cognition and contemplation are not predictions. Also the experience of higher consciousness is non-sensory and non-predictive. We also have the subconscious mind running our lives (and helping us catch balls) and the preconscious, which Freud posited and is often overlooked, but an important human experience based on my own observations. Keep up the good work, Suzi!
Thanks, Buck! I’m really looking forward to getting into this topic with you. I think you make some strong points, but I’m not entirely convinced that cognition and contemplation are completely non-predictive. Take contemplation, for instance — aren’t we essentially 'running simulations' during the process? It feels like we’re making predictions about various scenarios or outcomes. While these predictions may be more abstract, they still seem to be predictions in a sense.
That said, you’ve touched on something really crucial about the unconscious mind. What’s fascinating is that it seems we can make (at least low-level perceptual) predictions even when we’re unconscious. This does suggest that prediction and conscious experience are not the same thing, which is such an intriguing idea to unpack. I'll get into that research and question later in the year. Looking forward to reading your thoughts on it.
Years ago I participated in an international discussion group on the topic of Complexity Science. Everyone passionately expressed their point of view, and no one agreed. This led me to the principle I call IADOYPOV - It All Depends On Your Point Of View. This is not relativism but an acknowledgement that your philosophical starting point determines your perception on any topic. As a former Psych major, I love the idea of constructionism which Barrett et al. advocate, but it has limits. There are also people who believe that consciousness occurs outside the body, or as one psychic put it, consciousness just sticks its toe into the body. Relatedly, "I think therefore I am" is not the same as "I think therefore I predict." As I sit here I am thinking about what to type next (predicting). But as I stop and look at my right hand, just look at it, I am not predicting. I am observing, perceiving. At least I think so. LOL!
That sounds like a fun discussion group to be involved in! I do like the 'It All Depends On Your Point Of View' idea -- I also think it depends on the questions you are asking.
Absolutely! The questions you are asking are an outgrowth of your IADOYPOV, as well as your present circumstances. Keep up the great work Suzi!
Thanks, Buck!
I remember as a youth playing softball and catching at line drive hit directly at me. The only thing I remembered was the ball hitting my glove. Consciousness isn't all that involved in catching a ball once the skill is learned, it is heavily involved, however, in learning the skill in the first place.
Yes, I think you’ve hit on something really important here about the relationship between conscious learning and unconscious skill. I think it’s fascinating how, once we’ve mastered a skill, thinking too consciously about it can actually disrupt performance. But what I find particularly intriguing is why consciousness is even necessary during the learning phase. Why does the process of acquiring a skill seem to require this deliberate, conscious focus in the first place?
If consciousness is the mechanism for information integration (the mechanism NOT information itself as in IIT), then learning a skill would recruit information from many different parts of the brain. However, most of that information isn't required for the task. For example, you might start with 50 variables in a predictive model, but discover only 5 of them are needed to predict successfully.. Once the 5 are learned and refined, you can drop the other variables from the model. The high degree of information integration is no longer needed. The brain can do the task with less energy, always as important consideration evolutionarily since the brain demands for energy must be balanced with the body's demands.
The bigger the brain, the more refined the senses, the greater the requirements for integration across the brain.
Enjoy your break! Great short essay series. Thanks Suzi.
Thank you, John!
When first starting out playing catch, you do get hit in the face. Over time, your entire nervous and musculature system shorten the reaction time -- and no doubt shorten the detection-reaction circuit pathway. Consider all the evidence in the natural world where reaction exceeds conscious consideration: a cat dodging a snake bite, a dragonfly shifting to snatch a mosquito, a falcon striking a pigeon from the air. Tens of milliseconds. I can't imagine much conscious "thinking" going on there.
Yes, exactly! Those lightning-fast reactions you describe — like a cat dodging a snake or a dragonfly catching its prey — happen incredibly quickly. The brain is fascinating in this regard. Visual information related to movement is processed in the dorsal "where" pathway, while information about objects is processed in the ventral "what" pathway. Interestingly, the "where" pathway (movement) is faster than the "what" pathway (object recognition). For creatures like cats, the "where" pathway is even faster than ours, giving them a significant edge in reaction speed.
If the brain operated as a purely bottom-up system — with information flowing in only one direction — we would always move before becoming aware of what we were reacting to. However, the brain is far from purely bottom-up. There are countless feedback projections that influence, or bias, the incoming information. This interplay between feedforward and feedback processes makes the question of when things happen in the brain incredibly complex and far from straightforward.
Cool
So much depth there, we’re just dipping our toes.
One thing that blows my mind still is that there is not one brain that makes predictions but thousands of cortical columns (think, CPU cores) that each process the stimuli in predictive fashion and then *vote* by comparing their results laterally. 🤯
Exactly! It is mind-boggling! We often talk about "the brain" as if it’s a single, unified processor, but as you’ve pointed out, it’s more like a massive parallel processing system, with different parts making their own “decisions” simultaneously. It’s almost as if the brain is made up of thousands of tiny machines, each working independently yet in coordination.
Thanks for highlighting this — it’s such a crucial point. And you’re absolutely right — we’re only just dipping our toes in!
Seems you’ve dug out (pun intended) the core of baseball—pitchers with curve balls, sliders, screw balls, etc. against batters who are gaining experience during each at bat. I’m not sure we always predict but sometimes wait to see what happens or try to figure out what happens before taking action (so I question the idea that this is the core to consciousness).
We probably don’t recall getting hit in the face as babies but all normally developing children smack themselves in the face and then cry about the question who did this to them? Eventually some control develops with the growing self preservation instinct/drive. Newborns don’t catch balls tossed at them.
I recall one instance when my wife and I were waiting at a signaled intersection to cross a street. We were standing there at the curb waiting with our first infant daughter in her stroller. Suddenly we heard car tires screeching to our left! Instantly and without thinking I grabbed the stroller handle and ran backwards away from the curb and street corner—my wife who had one hand on the stroller stood at the curb frozen. Then I saw a car sliding sideways heading toward us—her! I yelled—MOVE! The car screeched to a halt and the car behind it slammed into the back of it. The light had changed and the first driver didn’t want to run the light—apparently—and the car behind didn’t expect this sudden stop. When all the cars had stopped my wife turned and yelled at me for grabbing the stroller so quickly and I yelled back what was she doing standing at the curb with screeching cars heading towards US?—a different us that we were just getting used to. Her answer—she was trying to figure out what was happening before she decided what to do (a puzzling but perfectly reasonable response). My point —who cares what is happening, run and get away then figure out what was going on (more the attitude stick your hand in front of your face first then see if a ball hits it—and if so catch it!).
And clearly the two drivers were using different method of living in the moment or predicting the next moment. That is, until their multiple methods converged—as did their cars.
Magical performance ( regardless of whether it is slight if hand, street magic, prop magic, or illusion) has a similar theme to create the effect—set the expectations and tempt the audience to get ahead of you, then do the unexpected. Magic!
Oh wow, Dean! I’m so glad you, your wife, and your baby were okay—that’s quite the story. You’ve captured such an interesting dynamic here: the contrast between instinctive, predictive reactions (like you with the stroller) and the more deliberate, analytical approach (like your wife).
I like your point about magic -- magicians succeed precisely because they understand how our predictive brains work. They set up our expectations, let our brains make predictions about what will happen next, and then violate those predictions. In a way, they're hacking our brain's prediction systems!
Thank you for sharing your ideas, Dean! You have such a knack for making abstract ideas feel vivid and relatable and, as a consequence, you make us think.
Very good question.
"This has led to a radical alternative theory: conscious experience is not a simulation of what is happening now -- it's a prediction of what will happen in the (very near) future."
Perhaps because of my long career in predictive analytics, I don't view the prediction idea as particularly radical. Maybe conscious experience is a mix of both a perception of what is happening now together with a prediction of what will happen in the very near future. Based on their experiences, people (and animals) are constantly refining their models of how the world works, and they use those models to predict the future, from near to distant. I'm reminded of this whenever I watch a dog skillfully track and catch a flying frisbee.
As an example of a longer range prediction, I recall some experiment involving placing a cat in a room in front of an opaque screen, and having a person walk from left to right, passing temporarily behind one side of the screen and emerging from the other side back into the cat's visual field. At some age cats have learned enough about the continuity of moving objects (and creatures) to expect the person to reappear, and if he/she doesn't the cat acts surprised and may walk behind the screen to try to figure out where the person went.
Your example of playing catch with your nephew reminds me of a day, several decades ago, when I was playing catch with a member of my brother's family (perhaps a nephew). I was backing up to catch a ball he had thrown, but didn't notice the rock or tree stump just behind me, so I tripped and fell on my rear end. I hardly felt any pain at the time, but over the next few days my neck, which had apparently taken a jolt from the fall, started getting sore. The discomfort got worse and worse until I thought I would need to seek medical attention. But just at that point it started getting better and cleared up completely over the next few days. I speculated that I had suffered some kind of minor "whiplash" injury.
Predicting the future also happens while we are reading. Based on the words we have read so far we narrow down our expectations for the next word or words, and those expectations actually alter what we think we read. This is one reason people make typographical mistakes while writing: while reading back what they have written, they tend to "fill in the blanks" with their expectations and don't even notice a missing or incorrect word.
Thanks, Dave! You've got me wanting to watch cat and dog YouTube.
I’m with you — prediction doesn’t feel all that radical. I spent about 10 years in a lab studying how attention and prediction shape our experience. So, when you mention your long career in predictive analytics, I can’t help but think — that sounds fascinating!
I love all your examples, but the reading one really stood out to me. It highlights how prediction happens at multiple levels. For instance, we predict at the letter level (reading misspelled words as if they’re correct), the word level (filling in missing words), and even at the meaning level (where two people can interpret the same text differently, predicting different outcomes or meanings).
You might think the biggest danger in playing catch is getting hit in the face with the ball — but nope! It's that sneaky tree stump lurking behind you! I'm glad you're okay.
Thanks, Suzi.
Your example "where two people can interpret the same text differently, predicting different outcomes or meanings" is relevant to the fields of law and diplomacy. Often, to get sufficient agreement to pass a law or ratify a treaty, its drafters must deliberately make the language sufficiently vague and ambiguous so the various stakeholders can interpret it to their satisfaction. But this is just "passing the buck". When later disputes arise about what the law or treaty actually means in practice, those ambiguities must somehow be addressed and resolved.
Here is an example of an embarrassing but amusing miscommunication that took place during the US presidency of Jimmy Carter. Per https://www.bbc.com/culture/article/20150202-the-greatest-mistranslations-ever:
"Jimmy Carter knew how to get an audience to pay attention. In a speech given during the US President's 1977 visit to Poland, he appeared to express sexual desire for the then-Communist country. Or that's what his interpreter said, anyway. It turned out Carter had said he wanted to learn about the Polish people's 'desires for the future'."
The BBC article cited a few more mistakes by his interpreters during that visit, and concluded: "By the time Carter's trip ended, he had become the punchline for many a Polish joke."
A little ambiguity can be strategic — until it isn't. Carter's interpreter took ‘lost in translation’ to a whole new level!
That reminds me of Sofia Coppola's entertaining film "Lost in Translation" (2003), starring Bill Murray and Scarlett Johansson. Per Wikipedia, "The film explores themes of alienation and disconnection against a backdrop of cultural displacement in Japan.".
Great movie!
Good description. Predictive processing is a good model which I think is correct but incomplete. In my own view there is a need for a Marvel Universe version where the Hypotheses Assemble. My pattern matching model, predictive processing, Global workspace theory and maybe some IIT as topping.
Good point! I agree that focusing too much on one theory can be a mistake. Too often, we get stuck in our own frameworks and overlook alternative perspectives simply because they are not 'our theory'.
Perhaps consciousness is not like a camera? Some analogies lead to new discoveries, some don't. The analogy that consciousness is like a camera leads to perplexities. Why retain it?
This goes towards explaining why in professional baseball, a batting average of .300 (three hits out of ten tries) is considered very good. Fastballs in the 90+ MPH range give the batter only 400 milliseconds to react. And while it does seem some top-level players actually can see the ball, there is an element of guesswork involved for most batters. (This is exactly why sign-stealing can be productive for a cheating team. Gives the batter a clue of what's coming.)
I had to look up 'sign-stealing'. Yeah, that would help — especially when you only have 400 milliseconds to decide whether to swing or let it fly past. Sign-stealing puts a new spin (puns!) on what we mean by a calculated guess.
It gives one an appreciation for the level of skill at the major league level. Most of us wouldn’t even see the ball fly past.
Teams have come up with elaborate schemes, binoculars and a walkie talkie in the bleachers, and someone in the dugout making sounds the batter can hear, but sometimes, with a runner on second (who can see the catcher’s signs) it’s just a matter of something like adjusting the ball cap for a fastball or adjusting the belt for a curve or whatever. Anything for an edge!