I think you're right to raise the question of what we really mean by "intelligence." "General intelligence" is too loosely defined to be useful beyond theoretical discussion. "Artificial intelligence" is almost as bad. When people say "AI," they usually (but not always) mean "Artificial Neural Network" or "ANN."
I actually happen to be developing an ANN processor which, as you describe, "grows" by changing the geometry of the neural network during the learning process. As far as I know, this is something that the likes of Torch and TensorFlow are not capable of. That being said though, I'm fairly confident that it won't be able to achieve any form of higher intelligence akin to the human brain, nor would it be considered "AGI."
I think that there are just some aspects of the human brain that simply can't be replicated by any machine; and I think that's quite beautiful honestly.
To answer Suzi's question, I think the first AGI will need to 'build' itself...in a similar way that infants lay on their backs, waving their arms, slow to realize that those are their arms, that they can control their arms and that said arms have affordances. I think it all needs to begin at that level. And it needs to happen without symbolic representations. After the first AGI, we can clone the rest. As each clone is released into the world, it will develop its own unique personality.
I'm inclined to agree with Tom (with the exception of point 4). Here's another thing that AGI wouldn't be able to that human brains can: simply exist. Not only does AGI not actually exist, but nobody even has any idea how it would work, even in theory. Most people think of AGI as this vague idea of "the next step beyond AI", but that's it. Other than that, nobody has ever managed to even define what AGI is, let alone build it.
I wonder why there is so much vagueness around AGI. Maybe it's because we're still trying to conceptualise intelligence. As you say, we often conceptualise AGI as human intelligence but faster. Maybe it's because we are still thinking about artificial intelligence through the lens of engineered systems. Perhaps. Why do you think we struggle to define it?
It seems to me that the idea of AGI comes from the original idea of what "artificial intelligence" used to mean in sci-fi movies. People once had this idea that AI was "basically a human in a computer" from movies like Space Odyssey with the HAL-9000. Once AI came to have the meaning closely associated with ANNs, people still had the original idea from the movie fantasy; thus, the term "AGI" was born. That's my speculation anyway; take it with a grain of salt.
There's also one interesting thought I have about attempting to replicate human intelligence on a computer. In the field of psychology, the fundamental question seems like "how does it work?" In computer science, that fundamental question is "what can we make it do?" or more specifically "what can be computed?"
Now, the question on people's minds is "can we compute human consciousness?" I'm inclined to say "no" for a couple reasons. First, in order to replicate something on a computer, you first must understand, on an intimate and technical level, what you're trying to replicate. We don't have such an understanding of the human mind, so I don't believe we could or should attempt to do that. Second, the brain and a computer are totally different systems, based on fundamentally different operating principles. Any attempt to replicate one system on the other is going to have issues, to put it lightly.
AI (ANNs in particular) is based on an interesting intersection between the field of psychology and computer science. We took a vague understanding of "how does it [the brain] work?", then filled in the blanks of what we didn't understand with "what can we make it do?". Thus, AI was born.
Hi Michael! You’re absolutely right—terminology in this area is incredibly vague. Terms like "general intelligence" and "artificial intelligence" have become so broad that they often lack precision. There’s definitely a need to clarify what we mean.
Your work on an ANN processor that can modify its network geometry during learning sounds fascinating! This kind of architectural plasticity feels much closer to how biological brains develop and adapt. I’d love to hear more about how it differs from traditional fixed-architecture ANNs.
I agree—architectural growth alone might not be enough for AGI. But I do wonder if it’s necessary. Flexible intelligence seems to rely on certain fundamental ingredients, and I can’t help but think that self-development might be one of them.
Thank you, Suzi! You do bring up a good point there. I reckon that anything with inflexible intelligence would only have a fixed set of capabilities and completely unable to evolve, thereby unable to be considered AGI. It makes sense to me to conclude that anything with "human-like" intelligence should have human-like characteristics, such as neuroplasticity. Though it seems to me that neuroplasticity would be only one of an innumerable list of characteristics that AGI would need to match in order to be "human-like".
I think another thing to consider is that intelligence probably doesn't HAVE to be human or human-like intelligence. AGI might turn out to be something very different from biological intelligence and either draw the same conclusions from a set of facts or draw some radically different conclusions, equally valid. It might be like introducing imaginary numbers or noneuclidean geometry. When the computers start building it, they may have very different ideas than their human predecessors.
That's a good point. That's kinda what already happened with AI, before our definition of AI became tied to ANNs. Might the same thing happen with AGI?
You seem to think that's the only way possible, right? I'm not sure, though. Does the fact that some AI systems are now passing the Turing Test mean anything here? https://substack.com/@tomrearick mentioned "neural plasticity;" In mechanical terms, how close is that to systemic redundancy? Couldn't something very close to neural plasticity be achieved by systemic redundancy?
Also, @tomrearick mentioned several other things he says brains can do that AI can't. Isn't one of the biggest questions around whether those things are emergent qualities? Self-guided learning, reasoning, meta-cognition and innovation spring to mind as possibly emergent. At what point does the "imitation" necessary to pass the Turing Test graduate into real thought? I'm inclined to think that point is very close and will happen in conjunction with robotics.
I don’t think systemic redundancy and neuroplasticity are similar at all. Systemic redundancy is simply operating two identical systems together, so if one fails, the other one can take over. Neuroplasticity is the idea that neurons change and evolve to enable more complex types of information processing.
Ultimately, our understanding of AGI is so incredibly vague, and doesn’t really exist outside of fantasy. If we do manage to nail down a more specific concept of AGI, I doubt it will be anything like what we think of it as currently.
That makes sense. Let me ask a question, then: is AI able to identify problems which it cannot solve? And if so, can it ask itself what it needs, in terms of programming, to be able to solve those problems? I thought the answers to both of those questions was "yes." If that's true, do you think it's analogous in function to neuroplasticity? I know that in some areas they're trying to introduce biological components, but for present purposes I'm assuming parts that cannot evolve and that all change will be in programming.
If cars were built like brains they'd be built doing 60 down the motorway (yes, I've seen the changing wheels video).
But computer programs aren't always completed before use. Corporate financial systems were (are?) usually developed as single entities (the finance dept don't like people having private "development" copies of the payroll program for some reason!) and they have to keep running while being further developed, eg to incorporate each year's tax changes. The way bugs briefly appear in Facebook suggests it's being developed on the fly too.
But if you knew what structure you needed to function as an AGI, you could build it, even if it needed to be self-extending.
I hadn’t seen the changing wheels video—what a great analogy!
Good point! Many complex software systems are developed and modified while running, especially large-scale enterprise and web applications.
But I wonder if there’s still a key difference between this kind of “live” software development and how biological brains develop. In software, we’re usually making planned modifications to a system with a pre-defined architecture. In contrast, the brain seems to design its own architecture through self-organisation and selection — more like a car assembling and redesigning itself while driving.
On your last point: if we did know the 'correct' structure for AGI, could we simply build it? Maybe. But given that we get biological intelligence through development and self-organisation, I wonder if we need to get the structure itself through a similar process rather than being fully designed in advance.
"the brain seems to design its own architecture" but then we end up with all brains looking remarkably similar. Even if you accept Henrich's apparently weakly founded view that people in WEIRD societies shift their facial recognition to the right hemisphere plus all the other changes he lists (page 3-4), we still end up with the same brain given the same environment. And of course we start with the same pre-defined architecture at birth. (probably we're just talking at different scales of difference)
re AGI: maybe we'd need to develop the first organically (how else would you engineer an emergent property?), but then we'd build thousands.
Yes, good point! Brains do all turn out pretty similar. I think this is true because not only does the process of building a brain involve self-organisation and plasticity, but it also follows programmed rules (developed through evolution). These rules seem to constrain what can and can't develop. I think of this as similar to how cellular automata like Rule 110 can produce complex but predictable patterns through simple iterative rules -- except with brains; environmental influences add an extra layer of complexity.
In a lot of contemporary software development, the agile methodology now mixes use and development. It's a tacit recognition that our ability to develop using the old waterfall method (documenting requirements, followed by specs, coding, testing, production, etc) was always far more limited than anyone wanted to acknowledge. Now minimum viable product is developed and put in front of the users, who provide feedback which is incorporated into further development. At some point it's good enough for real use, but the feedback-development cycle continues (usually for some contractually specified number of cycles or time period).
It's interesting that AI training hews closer to the old waterfall method. There's a lot to be said for being more "organic" about it, although I suspect there are technical challenges still to resolve to make it possible, such as allowing AI to update during use could lead to unpredictable results. The systems we have remain a lot more limited than the hype implies. Not that they won't eventually get there.
I've long found our reference to ourselves as a "general intelligence" somewhat dubious. It seems like we're a survival intelligence. Most of the fears of developing AI is that it will somehow turn into a survival intelligence of its own. But a survival intelligence seems like a very specific type of architecture. There are some dangers associated with tool intelligence, but the idea that it will accidentally become a survival intelligence seems similar to worrying that a word processor will accidentally become a travel reservation system.
I presume a lot of the push towards Agile is cos a major part of each system is a large corporate or other database that is already live and not easily (or legally) duplicatable (eg confidential data or live links to bank accounts)
(Of course it also helps to control mission creep. Under the Waterfall it was too easy to tempt the client into continual improvements until the money ran out then never have to deliver - look how many big government IT projects run over budget and get cancelled.)
It's actually about increasing the chances that the customer has a usable product at the end. Too many old school projects ran out of money or time with a system that wasn't ready yet. Agile, when done right, has the customer interacting with the system from a very early stage, which increases the chance they'll notice problems early and be able to get them corrected. Although it does depend on the customer being engaged.
Which isn't to say it's always done right. Or that things can't still go very wrong. And a lot of people just forgo requirements analysis and planning, shoot from the hip, and call what they're doing "agile". It's not unusual for those types of projects to turn into a mess.
I like your analogy with agile software development where developers learn as we go along. Existing ‘AI’ such as LLMs might be like a waterfall process where everything gets defined up front. But an AGI will learn as it goes along like an agile developer. While an LLM’s knowledge is fixed, an AGI will learn new information and new ways of acting as it interacts with the world.
If we think of an AGI learning new things all the time, I can see how it might learn that one day someone might want to turn it off and, essentially, kill it. The first AGI to learn that will have an advantage over other AIs and ‘survival’ will spread quickly — just like it did when life discovered that survival was important.
"The first AGI to learn that will have an advantage over other AIs and ‘survival’ will spread quickly"
Maybe, if we made them self replicating with a selection effect so that evolution kicked in. We probably should think very carefully before doing something like that.
That’s what I had in mind, yes. I agree that we should not rush to do this but we — you and I — are not the one likely to make billions of dollars making the first AGI so it doesn't really matter what we think.
"Maybe, if we made them self replicating with a selection effect so that evolution kicked in."
John Holland (https://en.wikipedia.org/wiki/John_Henry_Holland) was one of the pioneers in the development of "Genetic Algorithms", computer programs that maintain populations of digital "organisms" that learn from experience using analogs of human evolution such as mutation, cross-breeding, and survival of the fittest. I was inspired to write my first genetic algorithm in 1985 after attending a fascinating talk by Holland at a conference on "Evolution, games, and learning : models for adaptation in machines and nature".
I've had a long career and hobby as a computer programmer (I started in 1962!) with a specialty in predictive analytics. I've since experimented with other genetic algorithms, some based on my own designs and some based on open-source implementations for programming languages like Python, although the majority of my work (and play) in machine learning has been with decision tree ensembles, neural networks, and nearest neighbor algorithms. It's also possible to combine these in interesting ways, such as using genetic algorithms to evolve and optimize neural networks.
Excellent point! The distinction between survival intelligence and general intelligence captures what I was trying to express about different types of general intelligence. I agree -- survival-based intelligence, shaped by evolutionary pressures and innate motivations, would likely produce very different behaviours compared to machine intelligence. It’s hard to imagine a machine or tool without survival concerns accidentally developing the complex, adaptive behaviours we require for survival.
Interesting breakdown! The idea of human+ANN as an AGI raises some interesting questions about intelligence. I'll have to think about that one for a bit.
I'm guessing the standard AGI tests are highly oriented to a subset of human capabilities. At any rate, I thought I read AI is already performing at a 80-90% level AGI. Going past AGI may require AI to feed its own data back to itself, but that presents the challenge of subtle biases and idiosyncrasies becoming reenforced. The end result could be dangerous, useless, or both.
Yes, this is what I wonder about -- the flexibility that seems essential for AGI. We know biological intelligence develops through constant interaction with its environment, receiving feedback and corrections. The more complex the environment, the more sophisticated the system may become.
So perhaps for AI to evolve into what we would consider truly general intelligence, it would need to operate in dynamic, complex environments with continuous updating rather than existing as the kind of self-contained system we typically build today. The challenge then becomes not just about building the right architecture but creating a system that can grow and giving it the right conditions for flexible intelligence to emerge.
My understanding is that VERSES AI is trying to build what it calls a sentient AI. By its definition, it seems like "sentience" is a behavior of probing and interacting with the environment that does not necessarily imply consciousness.
Lovely essay, thank you. Having been damaged by the analytic school in my youth 😉, I’m loath to suggest that we’re probably not going to manage to discuss this in any resolvable manner using the current natural language around intelligence for a good long while. I don’t think that this necessarily militates against progress in the field at all though - but I suggest this progress will surely surprise most of us (and the language will follow).
It might be old hat to someone working closely within neuroscience, deep learning, etc., but it's an excellent layman-friendly breakdown that covers a lot of ground, including some of the "AGI building itself and turning into ASI very quickly" scenarios.
Wonderfully thought provoking as usual. What do you make of the new Chinese version of LLM? It seems to conform more closely to human intelligence in that the programming takes some of the load off the data crunching requirements of western LLM. It's far more efficient and, the article I read on it had this intriguing comment on it, "that it is less precise," which sounds like an opening for (human?) error.
I also note that the model you discuss explains how people can get smarter or dumber, in terms of real intellectual capacity, as they age. Both a promising and cautionary tale.
Thanks for another excellent piece of work. You really are great at this.
Thanks for your kind words, Jack! (And sorry for the slow response—it’s been a busy week.)
I haven’t looked too deeply into the new Chinese LLM, but I'm interested in the idea of a system that is more efficient but "less precise". That does sound like Daniel Kahneman's System 1 thinking, doesn't it!?. We often trade perfect precision for efficiency.
Yes, intelligence (as we often use the term) fluctuates over a lifetime, which is important to keep in mind. But because it’s so deeply tied to self-identity, that fluctuation can be a cautionary tale indeed!
I think the calculated trade-off of precision for efficiency in AI is a fascinating concept. I can see how it might work in LLM, but now I also have visions of calculating equations and giving the answer: “Oh, it’s about 427” or some back of the napkin type answer.
I can see how we might be drawn to this new type of model. The energy requirements for our typical current set-up are probably not sustainable. Efficiency might trump reliability in many cases. I guess the real test will be how it handles situations where reliability is critical. As you said, there’s a fine line between 'good enough' and 'not quite right'.
My experience with LLMs is quite limited, but I doubt more imprecision would be an issue there, since what I use now bears the same relation to what I need as a writing prompt does to a completed short story. Does LLM do anything that does require precision?
I guess some expect LLMs to be precise in specific domains like mathematical calculations, coding, or fact-checking. But, of course, their strength really lies in more open-ended tasks like brainstorming, exploring ideas, or (as you do) treating it as a writing prompt generator. In these cases, the level of precision may not matter much. I guess it's when people try to use LLMs for tasks requiring strict accuracy that precision becomes critical. But this seems like a problem with misunderstanding how they work more than an issue with the technology itself.
I actually use them for legal research, but really? Same difference! Sometimes you're looking for information on an abstract concept and just want to find the right ballpark for more precise inquiry, and LLMs seem pretty good at doing that.
As you say, we humans learn as we go along and I would expect an AGI to do the same. LLMs have separate ‘learning’ and ‘doing’ phases. I think AGI will also have an initial learning phase but unless the ‘AGI’ is actually learning as it goes along, it is not really intelligent at all.
I expect I agree completely with what you are saying but the framing just doesn't feel right to me.
Yes, I agree -- build itself isn't quite the correct framing. I think what I was trying to get at was the difference between engineered systems versus systems that grow and are self-organised. It's not just that biological brains learn as they go along, but it's that we get the very architecture through a process of growth -- where each step builds on previous steps, information unfolds over time, and the system self-organises in response to both internal and external factors.
I wonder how much the distinction between our current engineered systems and biological systems matters. It seems (at least to me) that growth allows for properties to emerge that may be difficult or impossible to engineer directly.
So, I think we do agree here. Perhaps the key question isn't whether AGI needs to "build itself" but rather how we might create conditions that allow for continuous development and adaptation.
I'm definitely a neophyte in this realm but my understanding of Animate Intelligence is simply an exchange of information; any information between organisms. My favorite example is a grove of trees exchanging information and nutrients via the mycelial network of fungal bodies.
Not unlike the FBI gathering bits of information that may or may not be useful but slowly building to useful data, though I guess what the FBI does being defined as useful is up to the individual.
I hope I'm not mistaken but that seems to me what Mr. Rearick is describing with the infant learning analogy.
That’s a really interesting way to think about it! It makes me question what we mean by information. What happens when this 'information' is exchanged? It seems that resource distribution and coordination would be involved, but this 'information' exchange also drives structural and functional change. In humans, this means physically reshaping the brain, which in turn affects perception, learning, and decision-making.
The question I’m interested in is whether the key difference between artificial and biological intelligence lies in how they change in response to information. AI systems update parameters or adjust weights, but biological intelligence reorganises itself, develops new structures, and integrates information in a way that fundamentally alters its capabilities over time.
Hi Suzi! This is my new Substack account. I’ll leave the rest going for a while at least as I switch over. I’ve only done an “About” essay so far here to let people know who I am and what my project happens to be. It’s pretty extensive though so I am satisfied for now.
Does general intelligence need to build itself given that our own general intelligence does that? This would be quite a coincidence however since there wasn’t any choice. All elements of evolution are self built.
My own suspicion is that general intelligence can only exist by means of a value dynamic. I theorize the brain essentially as a non-conscious energy driven computer that isn’t entirely unlike the non-conscious energy driven computers that we build. One output of the brain however is to create a computer that isn’t driven by energy, but rather by value. This is the consciousness that actually is us. Consciousness functions by interpreting inputs and constructing scenarios about what to do given its need to feel good rather than bad from moment to moment. So for us to build a true general intelligence we’d need to build something that feels good/bad. And of course I suspect the physics of value resides within the electromagnetic spectrum. But it’s not like we’d see an intelligence just because we create something that feels good/bad. Far more would be needed to make it even slightly functional. Because most today seem to think that the more something seems conscious, the more it is conscious, we trick ourselves with all sorts of AI Armageddon nonsense.
You raise an interesting question about causality. If I understand you correctly, you're suggesting that consciousness and value systems drive general intelligence, with the brain acting as a "non-conscious computer" that generates consciousness to make value-based decisions.
However, we might want to question the direction of causality here. Since the brain develops through physical growth over time, it could be that the brain's self-building process creates the conditions necessary for value-based decision-making to emerge. In other words, value systems and consciousness might be products of the developmental process rather than the drivers of it.
It seems like you’ve got the gist of what I’m saying reasonably well Suzi. So you wonder if it could go the other way? In theory, sure. That’s just not the direction I propose. But it’s certainly fine if you’re going that way right now. Actually while reading some of @tomrearick ’s stuff it finally dawned on me that like him, lately you seem to be going quite “embodied cognition”. As far as I can tell there are many flavors, though for the most part they’re probably closer to me than any of the other popularly marketed themes. The only thing that concerns me is that sometimes proponents get a little spooky with their deference to biology by not fully acknowledging the primacy of physics. If we understood the physics that the brain uses to create consciousness, then consciousness must be on the table technologically too. Conversely saying that only biology can do the job is kind of like saying that biology is the right kind of magic. But I have no indication that Tom gets spooky like that, and the little that I have looked at over there seems quite interesting.
Anyway the reason that I go the direction I do is because it corresponds with my general stance on psychology. Here value drives conscious brain function to thus have causal effects, like chosen muscle operation. If value were a product of development processes then there’d still need to be an explanation regarding how this by-product also goes on to create agency.
I did submit a first basic post since my original comment here, and it theorizes the means by which evolution causes continuous agency over time given that technically the agent can only be concerned about the now. Next I’ll get into a basic idea that seems to be missing in from academia, or the value of exiting in itself. Technically this resides under the domain of philosophy, though apparently philosophers have always been too preoccupied with the supposed rightness to wrongness of human behavior to open this matter up. And how might psychologists effectively explore the function of value driven life, if they don’t formally acknowledging a value to existing in itself?
> the only thing that concerns me is that sometimes proponents get a little spooky
Yes, I agree. Extreme versions of embodied cognition get very spooky indeed!
It also seems that your direction -- starting with value/agency rather than deriving them from development -- provides for the possibility of genuine free will. If agency emerges solely from developmental processes, it's harder to accept we have libertarian free will.
Your diplomacy is never required with me Suzi, but appreciated nonetheless. Furthermore you give me the opportunity to present some quite basic elements of what I believe. I’ll get to agency in a moment, but yes value is quite foundational for me. For example I think Descartes’ most famous line could have technically been improved as “I [experience value], therefore I am”. Though perhaps most of reality exists oblivious to its own existence, whatever does experience its existence should exclusively be in terms of a goodness to badness that’s felt. When value stops or starts, consciousness should stop or start. I consider them effectively the same.
There’s also Thomas Nagel’s “something it is like” heuristic for consciousness. Some? What thing might that “some” be? Perhaps validity was provided by an implicitly understood value dynamic without an explicit grasp that this is what was always doing the work? The value of existing is the topic of my coming post #2.
Though personal value is the fundamental beginning I think we all should discover in the end if we look hard enough, our sense of agency should ultimately reduce back to that, a “sense of”. And indeed, if captured and confined to a high enough degree a person’s sense of agency might be taken away to ultimately result in grave psychosis.
In the early 90s as a college kid there’s an agency perspective that struck me that I consider so effective, it perplexes me that I don’t see it from anyone else. I wonder if you’re familiar with the following idea Suzi? It’s essentially that all of reality is perfectly deterministic thus mandating that freewill cannot exist for us in an ultimate sense, but because we have extremely limited perspectives we feel like we’re free as a function of our vast ignorance. Observe that the more we learn about the circumstances that created someone else, the more our perception of agency based “goodness to evilness” should devolve back to a non agency based “goodness to badness”. I wonder if you’ve come across this reduction? And if anyone prominent has ever has made it, then what’s wrong with it such that naturalists today generally still seem confused?
As you point out, LLMs already need to be trained. This is because they start as blank slates. Gobs of input data and lots of compute time progressively tweak their parameters to create that multi-dimensional parameter space. While LLMs aren't brains, they are inspired by them, and there are similarities. For AGI, I suspect one does have to start with a blank slate network and train it.
If human brains are one end of a spectrum, and LLMs are the other, it seems AGI would have to live on that spectrum (the fear being it'll become the new end of the spectrum and be far beyond us). So, yeah, I think training will turn out to be necessary. The thing about AGI is that, once trained, it could presumably be easily duplicated and commodified.
I've read several SF books now in which AGI did require a real-world training period just like we do. In one case, that included emotional learning. The AGI was a young teenaged jerk. :)
I think you're right to raise the question of what we really mean by "intelligence." "General intelligence" is too loosely defined to be useful beyond theoretical discussion. "Artificial intelligence" is almost as bad. When people say "AI," they usually (but not always) mean "Artificial Neural Network" or "ANN."
I actually happen to be developing an ANN processor which, as you describe, "grows" by changing the geometry of the neural network during the learning process. As far as I know, this is something that the likes of Torch and TensorFlow are not capable of. That being said though, I'm fairly confident that it won't be able to achieve any form of higher intelligence akin to the human brain, nor would it be considered "AGI."
I think that there are just some aspects of the human brain that simply can't be replicated by any machine; and I think that's quite beautiful honestly.
Can you say more, Michael? What is an example of something human brains can do that an AGI won't be able to do? And why is that?
Here are nine things that animals (and humans) do but current AI foundation models cannot do and why AI will never scale into AGI.
1. Self-guided learning
2. Adaptation to Change
3. One-Shot, Feed-Forward Learning
4. Neural (architectural) plasticity
5. Size, Weight, and Power
6. Meta-cognition
7. Innovation
8. Reasoning
9. Emergence of Peircean Semiotics
Each is elaborated on at https://tomrearick.substack.com/p/ai-reset. To reach AGI, we will need to start over.
To answer Suzi's question, I think the first AGI will need to 'build' itself...in a similar way that infants lay on their backs, waving their arms, slow to realize that those are their arms, that they can control their arms and that said arms have affordances. I think it all needs to begin at that level. And it needs to happen without symbolic representations. After the first AGI, we can clone the rest. As each clone is released into the world, it will develop its own unique personality.
I'm inclined to agree with Tom (with the exception of point 4). Here's another thing that AGI wouldn't be able to that human brains can: simply exist. Not only does AGI not actually exist, but nobody even has any idea how it would work, even in theory. Most people think of AGI as this vague idea of "the next step beyond AI", but that's it. Other than that, nobody has ever managed to even define what AGI is, let alone build it.
I wonder why there is so much vagueness around AGI. Maybe it's because we're still trying to conceptualise intelligence. As you say, we often conceptualise AGI as human intelligence but faster. Maybe it's because we are still thinking about artificial intelligence through the lens of engineered systems. Perhaps. Why do you think we struggle to define it?
It seems to me that the idea of AGI comes from the original idea of what "artificial intelligence" used to mean in sci-fi movies. People once had this idea that AI was "basically a human in a computer" from movies like Space Odyssey with the HAL-9000. Once AI came to have the meaning closely associated with ANNs, people still had the original idea from the movie fantasy; thus, the term "AGI" was born. That's my speculation anyway; take it with a grain of salt.
There's also one interesting thought I have about attempting to replicate human intelligence on a computer. In the field of psychology, the fundamental question seems like "how does it work?" In computer science, that fundamental question is "what can we make it do?" or more specifically "what can be computed?"
Now, the question on people's minds is "can we compute human consciousness?" I'm inclined to say "no" for a couple reasons. First, in order to replicate something on a computer, you first must understand, on an intimate and technical level, what you're trying to replicate. We don't have such an understanding of the human mind, so I don't believe we could or should attempt to do that. Second, the brain and a computer are totally different systems, based on fundamentally different operating principles. Any attempt to replicate one system on the other is going to have issues, to put it lightly.
AI (ANNs in particular) is based on an interesting intersection between the field of psychology and computer science. We took a vague understanding of "how does it [the brain] work?", then filled in the blanks of what we didn't understand with "what can we make it do?". Thus, AI was born.
Hi Michael! You’re absolutely right—terminology in this area is incredibly vague. Terms like "general intelligence" and "artificial intelligence" have become so broad that they often lack precision. There’s definitely a need to clarify what we mean.
Your work on an ANN processor that can modify its network geometry during learning sounds fascinating! This kind of architectural plasticity feels much closer to how biological brains develop and adapt. I’d love to hear more about how it differs from traditional fixed-architecture ANNs.
I agree—architectural growth alone might not be enough for AGI. But I do wonder if it’s necessary. Flexible intelligence seems to rely on certain fundamental ingredients, and I can’t help but think that self-development might be one of them.
Thank you, Suzi! You do bring up a good point there. I reckon that anything with inflexible intelligence would only have a fixed set of capabilities and completely unable to evolve, thereby unable to be considered AGI. It makes sense to me to conclude that anything with "human-like" intelligence should have human-like characteristics, such as neuroplasticity. Though it seems to me that neuroplasticity would be only one of an innumerable list of characteristics that AGI would need to match in order to be "human-like".
Yes, I think you're onto something here.
I think another thing to consider is that intelligence probably doesn't HAVE to be human or human-like intelligence. AGI might turn out to be something very different from biological intelligence and either draw the same conclusions from a set of facts or draw some radically different conclusions, equally valid. It might be like introducing imaginary numbers or noneuclidean geometry. When the computers start building it, they may have very different ideas than their human predecessors.
That's a good point. That's kinda what already happened with AI, before our definition of AI became tied to ANNs. Might the same thing happen with AGI?
You seem to think that's the only way possible, right? I'm not sure, though. Does the fact that some AI systems are now passing the Turing Test mean anything here? https://substack.com/@tomrearick mentioned "neural plasticity;" In mechanical terms, how close is that to systemic redundancy? Couldn't something very close to neural plasticity be achieved by systemic redundancy?
Also, @tomrearick mentioned several other things he says brains can do that AI can't. Isn't one of the biggest questions around whether those things are emergent qualities? Self-guided learning, reasoning, meta-cognition and innovation spring to mind as possibly emergent. At what point does the "imitation" necessary to pass the Turing Test graduate into real thought? I'm inclined to think that point is very close and will happen in conjunction with robotics.
I don’t think systemic redundancy and neuroplasticity are similar at all. Systemic redundancy is simply operating two identical systems together, so if one fails, the other one can take over. Neuroplasticity is the idea that neurons change and evolve to enable more complex types of information processing.
Ultimately, our understanding of AGI is so incredibly vague, and doesn’t really exist outside of fantasy. If we do manage to nail down a more specific concept of AGI, I doubt it will be anything like what we think of it as currently.
That makes sense. Let me ask a question, then: is AI able to identify problems which it cannot solve? And if so, can it ask itself what it needs, in terms of programming, to be able to solve those problems? I thought the answers to both of those questions was "yes." If that's true, do you think it's analogous in function to neuroplasticity? I know that in some areas they're trying to introduce biological components, but for present purposes I'm assuming parts that cannot evolve and that all change will be in programming.
Yes, good point!
Another beautiful piece Suzi!
If cars were built like brains they'd be built doing 60 down the motorway (yes, I've seen the changing wheels video).
But computer programs aren't always completed before use. Corporate financial systems were (are?) usually developed as single entities (the finance dept don't like people having private "development" copies of the payroll program for some reason!) and they have to keep running while being further developed, eg to incorporate each year's tax changes. The way bugs briefly appear in Facebook suggests it's being developed on the fly too.
But if you knew what structure you needed to function as an AGI, you could build it, even if it needed to be self-extending.
Thanks, Malcolm!
I hadn’t seen the changing wheels video—what a great analogy!
Good point! Many complex software systems are developed and modified while running, especially large-scale enterprise and web applications.
But I wonder if there’s still a key difference between this kind of “live” software development and how biological brains develop. In software, we’re usually making planned modifications to a system with a pre-defined architecture. In contrast, the brain seems to design its own architecture through self-organisation and selection — more like a car assembling and redesigning itself while driving.
On your last point: if we did know the 'correct' structure for AGI, could we simply build it? Maybe. But given that we get biological intelligence through development and self-organisation, I wonder if we need to get the structure itself through a similar process rather than being fully designed in advance.
"the brain seems to design its own architecture" but then we end up with all brains looking remarkably similar. Even if you accept Henrich's apparently weakly founded view that people in WEIRD societies shift their facial recognition to the right hemisphere plus all the other changes he lists (page 3-4), we still end up with the same brain given the same environment. And of course we start with the same pre-defined architecture at birth. (probably we're just talking at different scales of difference)
re AGI: maybe we'd need to develop the first organically (how else would you engineer an emergent property?), but then we'd build thousands.
Yes, good point! Brains do all turn out pretty similar. I think this is true because not only does the process of building a brain involve self-organisation and plasticity, but it also follows programmed rules (developed through evolution). These rules seem to constrain what can and can't develop. I think of this as similar to how cellular automata like Rule 110 can produce complex but predictable patterns through simple iterative rules -- except with brains; environmental influences add an extra layer of complexity.
In a lot of contemporary software development, the agile methodology now mixes use and development. It's a tacit recognition that our ability to develop using the old waterfall method (documenting requirements, followed by specs, coding, testing, production, etc) was always far more limited than anyone wanted to acknowledge. Now minimum viable product is developed and put in front of the users, who provide feedback which is incorporated into further development. At some point it's good enough for real use, but the feedback-development cycle continues (usually for some contractually specified number of cycles or time period).
It's interesting that AI training hews closer to the old waterfall method. There's a lot to be said for being more "organic" about it, although I suspect there are technical challenges still to resolve to make it possible, such as allowing AI to update during use could lead to unpredictable results. The systems we have remain a lot more limited than the hype implies. Not that they won't eventually get there.
I've long found our reference to ourselves as a "general intelligence" somewhat dubious. It seems like we're a survival intelligence. Most of the fears of developing AI is that it will somehow turn into a survival intelligence of its own. But a survival intelligence seems like a very specific type of architecture. There are some dangers associated with tool intelligence, but the idea that it will accidentally become a survival intelligence seems similar to worrying that a word processor will accidentally become a travel reservation system.
Excellent post Suzi!
I presume a lot of the push towards Agile is cos a major part of each system is a large corporate or other database that is already live and not easily (or legally) duplicatable (eg confidential data or live links to bank accounts)
(Of course it also helps to control mission creep. Under the Waterfall it was too easy to tempt the client into continual improvements until the money ran out then never have to deliver - look how many big government IT projects run over budget and get cancelled.)
It's actually about increasing the chances that the customer has a usable product at the end. Too many old school projects ran out of money or time with a system that wasn't ready yet. Agile, when done right, has the customer interacting with the system from a very early stage, which increases the chance they'll notice problems early and be able to get them corrected. Although it does depend on the customer being engaged.
Which isn't to say it's always done right. Or that things can't still go very wrong. And a lot of people just forgo requirements analysis and planning, shoot from the hip, and call what they're doing "agile". It's not unusual for those types of projects to turn into a mess.
I like your analogy with agile software development where developers learn as we go along. Existing ‘AI’ such as LLMs might be like a waterfall process where everything gets defined up front. But an AGI will learn as it goes along like an agile developer. While an LLM’s knowledge is fixed, an AGI will learn new information and new ways of acting as it interacts with the world.
If we think of an AGI learning new things all the time, I can see how it might learn that one day someone might want to turn it off and, essentially, kill it. The first AGI to learn that will have an advantage over other AIs and ‘survival’ will spread quickly — just like it did when life discovered that survival was important.
Thanks.
"The first AGI to learn that will have an advantage over other AIs and ‘survival’ will spread quickly"
Maybe, if we made them self replicating with a selection effect so that evolution kicked in. We probably should think very carefully before doing something like that.
That’s what I had in mind, yes. I agree that we should not rush to do this but we — you and I — are not the one likely to make billions of dollars making the first AGI so it doesn't really matter what we think.
"Maybe, if we made them self replicating with a selection effect so that evolution kicked in."
John Holland (https://en.wikipedia.org/wiki/John_Henry_Holland) was one of the pioneers in the development of "Genetic Algorithms", computer programs that maintain populations of digital "organisms" that learn from experience using analogs of human evolution such as mutation, cross-breeding, and survival of the fittest. I was inspired to write my first genetic algorithm in 1985 after attending a fascinating talk by Holland at a conference on "Evolution, games, and learning : models for adaptation in machines and nature".
I've had a long career and hobby as a computer programmer (I started in 1962!) with a specialty in predictive analytics. I've since experimented with other genetic algorithms, some based on my own designs and some based on open-source implementations for programming languages like Python, although the majority of my work (and play) in machine learning has been with decision tree ensembles, neural networks, and nearest neighbor algorithms. It's also possible to combine these in interesting ways, such as using genetic algorithms to evolve and optimize neural networks.
Excellent point! The distinction between survival intelligence and general intelligence captures what I was trying to express about different types of general intelligence. I agree -- survival-based intelligence, shaped by evolutionary pressures and innate motivations, would likely produce very different behaviours compared to machine intelligence. It’s hard to imagine a machine or tool without survival concerns accidentally developing the complex, adaptive behaviours we require for survival.
Take AI to mean a trained ANN like an LLM and AGI to mean a system which will build (eg) an LLM when it needs it.
The AGI needs to
1. understand its environment, at least sufficient for 2-7
2. have (one or more) aims,
3. be capable of actions (eg build an LLM, move its arm),
4. know its capabilities, at least sufficient for 5-7
5. invent combinations of actions,
6. evaluate how these combinations further its aims,
7. put a combination into effect.
We automatically train our neural net whenever we repeat an action. The AGI could be more aware of this and turn training on or off.
Interestingly individual humans are AGI's but human team+ANN is also an AGI.
Interesting breakdown! The idea of human+ANN as an AGI raises some interesting questions about intelligence. I'll have to think about that one for a bit.
I proposed a Naked and Afraid test for AI once.
https://broadspeculations.com/2023/12/21/naked-and-afraid-ai-test/
I'm guessing the standard AGI tests are highly oriented to a subset of human capabilities. At any rate, I thought I read AI is already performing at a 80-90% level AGI. Going past AGI may require AI to feed its own data back to itself, but that presents the challenge of subtle biases and idiosyncrasies becoming reenforced. The end result could be dangerous, useless, or both.
I think to count as AGI, the AI has to be able to learn new tricks. As Suzi says, we are still teaching them everything upfront at the moment.
Yes, this is what I wonder about -- the flexibility that seems essential for AGI. We know biological intelligence develops through constant interaction with its environment, receiving feedback and corrections. The more complex the environment, the more sophisticated the system may become.
So perhaps for AI to evolve into what we would consider truly general intelligence, it would need to operate in dynamic, complex environments with continuous updating rather than existing as the kind of self-contained system we typically build today. The challenge then becomes not just about building the right architecture but creating a system that can grow and giving it the right conditions for flexible intelligence to emerge.
My understanding is that VERSES AI is trying to build what it calls a sentient AI. By its definition, it seems like "sentience" is a behavior of probing and interacting with the environment that does not necessarily imply consciousness.
https://www.verses.ai/genius
Lovely essay, thank you. Having been damaged by the analytic school in my youth 😉, I’m loath to suggest that we’re probably not going to manage to discuss this in any resolvable manner using the current natural language around intelligence for a good long while. I don’t think that this necessarily militates against progress in the field at all though - but I suggest this progress will surely surprise most of us (and the language will follow).
Thanks for that nice and clear-cut distinction, Suzi.
If you haven't already, I highly recommend this---*checks watch*---10-year-old piece (!) by Tim Urban in two parts: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
It might be old hat to someone working closely within neuroscience, deep learning, etc., but it's an excellent layman-friendly breakdown that covers a lot of ground, including some of the "AGI building itself and turning into ASI very quickly" scenarios.
This is amazing! Thanks, Daniel :)
Wonderfully thought provoking as usual. What do you make of the new Chinese version of LLM? It seems to conform more closely to human intelligence in that the programming takes some of the load off the data crunching requirements of western LLM. It's far more efficient and, the article I read on it had this intriguing comment on it, "that it is less precise," which sounds like an opening for (human?) error.
I also note that the model you discuss explains how people can get smarter or dumber, in terms of real intellectual capacity, as they age. Both a promising and cautionary tale.
Thanks for another excellent piece of work. You really are great at this.
Thanks for your kind words, Jack! (And sorry for the slow response—it’s been a busy week.)
I haven’t looked too deeply into the new Chinese LLM, but I'm interested in the idea of a system that is more efficient but "less precise". That does sound like Daniel Kahneman's System 1 thinking, doesn't it!?. We often trade perfect precision for efficiency.
Yes, intelligence (as we often use the term) fluctuates over a lifetime, which is important to keep in mind. But because it’s so deeply tied to self-identity, that fluctuation can be a cautionary tale indeed!
I think the calculated trade-off of precision for efficiency in AI is a fascinating concept. I can see how it might work in LLM, but now I also have visions of calculating equations and giving the answer: “Oh, it’s about 427” or some back of the napkin type answer.
I can see how we might be drawn to this new type of model. The energy requirements for our typical current set-up are probably not sustainable. Efficiency might trump reliability in many cases. I guess the real test will be how it handles situations where reliability is critical. As you said, there’s a fine line between 'good enough' and 'not quite right'.
My experience with LLMs is quite limited, but I doubt more imprecision would be an issue there, since what I use now bears the same relation to what I need as a writing prompt does to a completed short story. Does LLM do anything that does require precision?
I guess some expect LLMs to be precise in specific domains like mathematical calculations, coding, or fact-checking. But, of course, their strength really lies in more open-ended tasks like brainstorming, exploring ideas, or (as you do) treating it as a writing prompt generator. In these cases, the level of precision may not matter much. I guess it's when people try to use LLMs for tasks requiring strict accuracy that precision becomes critical. But this seems like a problem with misunderstanding how they work more than an issue with the technology itself.
I actually use them for legal research, but really? Same difference! Sometimes you're looking for information on an abstract concept and just want to find the right ballpark for more precise inquiry, and LLMs seem pretty good at doing that.
> does it need to build itself?
That feels like an odd framing to me.
As you say, we humans learn as we go along and I would expect an AGI to do the same. LLMs have separate ‘learning’ and ‘doing’ phases. I think AGI will also have an initial learning phase but unless the ‘AGI’ is actually learning as it goes along, it is not really intelligent at all.
I expect I agree completely with what you are saying but the framing just doesn't feel right to me.
Yes, I agree -- build itself isn't quite the correct framing. I think what I was trying to get at was the difference between engineered systems versus systems that grow and are self-organised. It's not just that biological brains learn as they go along, but it's that we get the very architecture through a process of growth -- where each step builds on previous steps, information unfolds over time, and the system self-organises in response to both internal and external factors.
I wonder how much the distinction between our current engineered systems and biological systems matters. It seems (at least to me) that growth allows for properties to emerge that may be difficult or impossible to engineer directly.
So, I think we do agree here. Perhaps the key question isn't whether AGI needs to "build itself" but rather how we might create conditions that allow for continuous development and adaptation.
Thanks for helping me clarify this point.
I'm definitely a neophyte in this realm but my understanding of Animate Intelligence is simply an exchange of information; any information between organisms. My favorite example is a grove of trees exchanging information and nutrients via the mycelial network of fungal bodies.
Not unlike the FBI gathering bits of information that may or may not be useful but slowly building to useful data, though I guess what the FBI does being defined as useful is up to the individual.
I hope I'm not mistaken but that seems to me what Mr. Rearick is describing with the infant learning analogy.
That’s a really interesting way to think about it! It makes me question what we mean by information. What happens when this 'information' is exchanged? It seems that resource distribution and coordination would be involved, but this 'information' exchange also drives structural and functional change. In humans, this means physically reshaping the brain, which in turn affects perception, learning, and decision-making.
The question I’m interested in is whether the key difference between artificial and biological intelligence lies in how they change in response to information. AI systems update parameters or adjust weights, but biological intelligence reorganises itself, develops new structures, and integrates information in a way that fundamentally alters its capabilities over time.
Hi Suzi! This is my new Substack account. I’ll leave the rest going for a while at least as I switch over. I’ve only done an “About” essay so far here to let people know who I am and what my project happens to be. It’s pretty extensive though so I am satisfied for now.
Does general intelligence need to build itself given that our own general intelligence does that? This would be quite a coincidence however since there wasn’t any choice. All elements of evolution are self built.
My own suspicion is that general intelligence can only exist by means of a value dynamic. I theorize the brain essentially as a non-conscious energy driven computer that isn’t entirely unlike the non-conscious energy driven computers that we build. One output of the brain however is to create a computer that isn’t driven by energy, but rather by value. This is the consciousness that actually is us. Consciousness functions by interpreting inputs and constructing scenarios about what to do given its need to feel good rather than bad from moment to moment. So for us to build a true general intelligence we’d need to build something that feels good/bad. And of course I suspect the physics of value resides within the electromagnetic spectrum. But it’s not like we’d see an intelligence just because we create something that feels good/bad. Far more would be needed to make it even slightly functional. Because most today seem to think that the more something seems conscious, the more it is conscious, we trick ourselves with all sorts of AI Armageddon nonsense.
Thanks, Eric!
You raise an interesting question about causality. If I understand you correctly, you're suggesting that consciousness and value systems drive general intelligence, with the brain acting as a "non-conscious computer" that generates consciousness to make value-based decisions.
However, we might want to question the direction of causality here. Since the brain develops through physical growth over time, it could be that the brain's self-building process creates the conditions necessary for value-based decision-making to emerge. In other words, value systems and consciousness might be products of the developmental process rather than the drivers of it.
It seems like you’ve got the gist of what I’m saying reasonably well Suzi. So you wonder if it could go the other way? In theory, sure. That’s just not the direction I propose. But it’s certainly fine if you’re going that way right now. Actually while reading some of @tomrearick ’s stuff it finally dawned on me that like him, lately you seem to be going quite “embodied cognition”. As far as I can tell there are many flavors, though for the most part they’re probably closer to me than any of the other popularly marketed themes. The only thing that concerns me is that sometimes proponents get a little spooky with their deference to biology by not fully acknowledging the primacy of physics. If we understood the physics that the brain uses to create consciousness, then consciousness must be on the table technologically too. Conversely saying that only biology can do the job is kind of like saying that biology is the right kind of magic. But I have no indication that Tom gets spooky like that, and the little that I have looked at over there seems quite interesting.
Anyway the reason that I go the direction I do is because it corresponds with my general stance on psychology. Here value drives conscious brain function to thus have causal effects, like chosen muscle operation. If value were a product of development processes then there’d still need to be an explanation regarding how this by-product also goes on to create agency.
I did submit a first basic post since my original comment here, and it theorizes the means by which evolution causes continuous agency over time given that technically the agent can only be concerned about the now. Next I’ll get into a basic idea that seems to be missing in from academia, or the value of exiting in itself. Technically this resides under the domain of philosophy, though apparently philosophers have always been too preoccupied with the supposed rightness to wrongness of human behavior to open this matter up. And how might psychologists effectively explore the function of value driven life, if they don’t formally acknowledging a value to existing in itself?
> the only thing that concerns me is that sometimes proponents get a little spooky
Yes, I agree. Extreme versions of embodied cognition get very spooky indeed!
It also seems that your direction -- starting with value/agency rather than deriving them from development -- provides for the possibility of genuine free will. If agency emerges solely from developmental processes, it's harder to accept we have libertarian free will.
Your diplomacy is never required with me Suzi, but appreciated nonetheless. Furthermore you give me the opportunity to present some quite basic elements of what I believe. I’ll get to agency in a moment, but yes value is quite foundational for me. For example I think Descartes’ most famous line could have technically been improved as “I [experience value], therefore I am”. Though perhaps most of reality exists oblivious to its own existence, whatever does experience its existence should exclusively be in terms of a goodness to badness that’s felt. When value stops or starts, consciousness should stop or start. I consider them effectively the same.
There’s also Thomas Nagel’s “something it is like” heuristic for consciousness. Some? What thing might that “some” be? Perhaps validity was provided by an implicitly understood value dynamic without an explicit grasp that this is what was always doing the work? The value of existing is the topic of my coming post #2.
Though personal value is the fundamental beginning I think we all should discover in the end if we look hard enough, our sense of agency should ultimately reduce back to that, a “sense of”. And indeed, if captured and confined to a high enough degree a person’s sense of agency might be taken away to ultimately result in grave psychosis.
In the early 90s as a college kid there’s an agency perspective that struck me that I consider so effective, it perplexes me that I don’t see it from anyone else. I wonder if you’re familiar with the following idea Suzi? It’s essentially that all of reality is perfectly deterministic thus mandating that freewill cannot exist for us in an ultimate sense, but because we have extremely limited perspectives we feel like we’re free as a function of our vast ignorance. Observe that the more we learn about the circumstances that created someone else, the more our perception of agency based “goodness to evilness” should devolve back to a non agency based “goodness to badness”. I wonder if you’ve come across this reduction? And if anyone prominent has ever has made it, then what’s wrong with it such that naturalists today generally still seem confused?
As you point out, LLMs already need to be trained. This is because they start as blank slates. Gobs of input data and lots of compute time progressively tweak their parameters to create that multi-dimensional parameter space. While LLMs aren't brains, they are inspired by them, and there are similarities. For AGI, I suspect one does have to start with a blank slate network and train it.
If human brains are one end of a spectrum, and LLMs are the other, it seems AGI would have to live on that spectrum (the fear being it'll become the new end of the spectrum and be far beyond us). So, yeah, I think training will turn out to be necessary. The thing about AGI is that, once trained, it could presumably be easily duplicated and commodified.
I've read several SF books now in which AGI did require a real-world training period just like we do. In one case, that included emotional learning. The AGI was a young teenaged jerk. :)
SF loves an AGI that is a jerk!