The example of turbulence is apt. Kolmogorov’s theory and subsequent developments along those lines are the most we can probably expect. As noted in the post, useful results and devices (aircraft) can be designed with the current understanding. Computational fluid dynamics is somewhat helpful but most of the predictive value in the physics of fluids comes from semi-empirical and heuristic methods like dimensional analysis.
The trouble with turbulent fluids is not complexity as much as it is chaos. As Edward Lorenz showed back in the 1960s, extremely simple systems exhibit deterministic chaos. I suspect chaos plays a role in the brain, which means it does not lend itself to closed-form, analytic solutions. So what? Airplanes still fly and a human-designed GAI can be created. Take it from an older, albeit not distinguished, physicist.
It may be possible to build GAI. I suspect (not having read the book) that it is possible. That doesn't mean that current methods (i.e. LLMs and their close relatives) are the way to do it.
In fact from my understandng of how LLMs work I think that they are dead ends like the various analog computers developed in the early/mid twentieth century.
LLMs have turned out to be extremely useful given their basic architecture, emergent properties at their finest. But it's not how were going to get AGI
A chess or go engine can learn first by studying human games than improve by playing with itself. A chatbot can be trained on stuff scraped from the internet but it can't improve on that so it's stuck at redditor level. Google AI shows links for its sources which are frequently Reddit and Quora threads. At least it doesn't quote 4chan or Tumblr.
Very good review, thanks. I agree with your conclusions, the authors' arguments do seem like being close to the best of their ilk and they are indeed weak enough to reinforce the opposite view.
I will let you know if I ever succeed in writing my own steelman of the case against AGI, your review would be very helpful.
I note that there might be a small category error in your opening when you go from GAI to AI trained on special tasks, specifically with video games. It is my understanding from the last few months (so maybe out of date already) that the generalized AI models do pretty terribly at games, even chess and amusingly Pokemon, even though specialized AI do a lot better. I think that is important to keep in mind, as a GAI would be expected to be human level at any random task; hyper trained specialist AIs have been around for a bit, and are rather less interesting. If Claude or GPT could step in and play any game better than or equal to a human with just a few hours of practice that would be incredibly impressive, but my understanding is that they are severely limited in this way.
The Trackmania example and the games were not LLMs, but other ANNs. LLMs can play some games, but they aren't that great at Chess despite having 'read' every textbook on the topic. They cannot even make legal moves much of the time. This series was amusing. https://www.youtube.com/watch?v=-m33dn_3sNQ
OT: "no, Greeks didn't know about elements, Greek atoms are not the atoms of physics, since they are obviously divisible" Hellenic Era Greeks did not, Hellenistic Era Greeks probably did reach the John Dalton's level of understanding matter as made of atoms (then Romans killed them and lost the memos, re: The Forgotten Revolution: How Science Was Born in 300 BC and Why it Had to Be Reborn Russo L. ISBN-13 978-3540203964)
A quibble, but you referenced the dot.com bubble as an allusion to the notion that AI is "a fad", but the dot.com bubble comparison actually works very well precisely because it is not a fad.
The internet didn't disappear after the bubble burst, and its significance has grown beyond what most anticipated at the peak of the bubble. But it was also a bubble because of early irrational exhuberance.
I expect a lot of present-day "AI investment" to crash and burn, but I also expect LLMs (or their successors) to grow in significance over the long run.
This appears to be primarily an argument between people who believe humans have transcendental properties (including intelligence) and consistent materialists who assert they don’t. I take a weird middle ground: supernatural things exist but intelligence is purely material, so there’s no categorical reason why it couldn’t be recreated materially.
Usually, one uses RL instead of supervised learning because one has no choice, not as a matter of speed. As far as I know, it is everywhere and always the worse option. RL comprises (more or less by definiton) a set of methods for estimating gradients from data, via experience, which one must do when one is not provided with the correct targets for learning. In the case of trackmania, these would be the optimal actions given any state, which are unknown.
I think you were trying to point out the sparse reward vs dense reward issue. The "naive" reward structure of the task in trackmania is completion time, and that is a reward you see rarely relative to action selection (sparse). Including proxies for performance can make the reward structure denser and improve learning speed dramatically.
Edit: I now see that maybe you meant RL is a fast optimizer relative to something like an evolutionary method. Fair enough. This just tripped a wire for me because RL has very bad sample complexity, generally speaking.
We must keep in mind what computers are, and what humans are. They are different.
Computers are syntactic engines. They do syntax, and syntax only. They are incapable of "understanding" anything. They are incapable of doing semantics.
Humans, on the other hand, really "understand". Humans do semantics.
This is a major difference in capabilities, which will never be bridged, due to the nature of what creatures (we created them) computers are, and what creatures (God created us) humans are.
In addition, there *are* in fact fundamental limits, akin to the Shockley–Queisser limit, for AI. Gödel's incompleteness theorems, for example. Gödel destroyed Hilbert's dream of a completely axiomatized mathematics. This has major implications for the limitations of the capabilities of AI.
We live in an age where generative AI is making very impressive advancements, but it is important to keep in mind that GENERAL AI is not making such advancements. General AI is stagnant. According to Gödel's demolishing of Hilbert's dreams, General AI will never make much progress.
This is, partly, due to the difference between what kind of creatures computers are, and what kind of creatures humans are.
It should be self-evident to anybody who understands how computers work that they are syntax engines, and only syntax engines. It matters what symbols come in, in what order, and from what interfaces (keyboard, mouse, network, etc.) But they are all just (to the computer) meaningless symbols.
This symbolic manipulation is exactly what David Hilbert had in mind with his ambitious mathematical goals last century.
It should also be self-evident to any human who can understand Gödel's incompleteness theorems that humans really DO understand what Gödel put forth. Humans are doing semantics in that case.
So humans have a capability that computers do not have, and can NEVER have, if Gödel is correct.
The example of turbulence is apt. Kolmogorov’s theory and subsequent developments along those lines are the most we can probably expect. As noted in the post, useful results and devices (aircraft) can be designed with the current understanding. Computational fluid dynamics is somewhat helpful but most of the predictive value in the physics of fluids comes from semi-empirical and heuristic methods like dimensional analysis.
The trouble with turbulent fluids is not complexity as much as it is chaos. As Edward Lorenz showed back in the 1960s, extremely simple systems exhibit deterministic chaos. I suspect chaos plays a role in the brain, which means it does not lend itself to closed-form, analytic solutions. So what? Airplanes still fly and a human-designed GAI can be created. Take it from an older, albeit not distinguished, physicist.
Excellent points.
It may be possible to build GAI. I suspect (not having read the book) that it is possible. That doesn't mean that current methods (i.e. LLMs and their close relatives) are the way to do it.
In fact from my understandng of how LLMs work I think that they are dead ends like the various analog computers developed in the early/mid twentieth century.
LLMs have turned out to be extremely useful given their basic architecture, emergent properties at their finest. But it's not how were going to get AGI
A chess or go engine can learn first by studying human games than improve by playing with itself. A chatbot can be trained on stuff scraped from the internet but it can't improve on that so it's stuck at redditor level. Google AI shows links for its sources which are frequently Reddit and Quora threads. At least it doesn't quote 4chan or Tumblr.
The paper discussed here - https://arstechnica.com/ai/2025/08/researchers-find-llms-are-bad-at-logical-inference-good-at-fluent-nonsense/ - says that LLM is incapable of learning more after its initial training
ISTR that google's go engine has been able to develop successful new plays that no one has ever seen before
LLMs have Wiliams Syndrome, sort of. https://en.wikipedia.org/wiki/Williams_syndrome It's a very interesting condition.
Very good review, thanks. I agree with your conclusions, the authors' arguments do seem like being close to the best of their ilk and they are indeed weak enough to reinforce the opposite view.
I will let you know if I ever succeed in writing my own steelman of the case against AGI, your review would be very helpful.
Very interesting essay, thank you.
I note that there might be a small category error in your opening when you go from GAI to AI trained on special tasks, specifically with video games. It is my understanding from the last few months (so maybe out of date already) that the generalized AI models do pretty terribly at games, even chess and amusingly Pokemon, even though specialized AI do a lot better. I think that is important to keep in mind, as a GAI would be expected to be human level at any random task; hyper trained specialist AIs have been around for a bit, and are rather less interesting. If Claude or GPT could step in and play any game better than or equal to a human with just a few hours of practice that would be incredibly impressive, but my understanding is that they are severely limited in this way.
The Trackmania example and the games were not LLMs, but other ANNs. LLMs can play some games, but they aren't that great at Chess despite having 'read' every textbook on the topic. They cannot even make legal moves much of the time. This series was amusing. https://www.youtube.com/watch?v=-m33dn_3sNQ
Scaling
OT: "no, Greeks didn't know about elements, Greek atoms are not the atoms of physics, since they are obviously divisible" Hellenic Era Greeks did not, Hellenistic Era Greeks probably did reach the John Dalton's level of understanding matter as made of atoms (then Romans killed them and lost the memos, re: The Forgotten Revolution: How Science Was Born in 300 BC and Why it Had to Be Reborn Russo L. ISBN-13 978-3540203964)
A quibble, but you referenced the dot.com bubble as an allusion to the notion that AI is "a fad", but the dot.com bubble comparison actually works very well precisely because it is not a fad.
The internet didn't disappear after the bubble burst, and its significance has grown beyond what most anticipated at the peak of the bubble. But it was also a bubble because of early irrational exhuberance.
I expect a lot of present-day "AI investment" to crash and burn, but I also expect LLMs (or their successors) to grow in significance over the long run.
This appears to be primarily an argument between people who believe humans have transcendental properties (including intelligence) and consistent materialists who assert they don’t. I take a weird middle ground: supernatural things exist but intelligence is purely material, so there’s no categorical reason why it couldn’t be recreated materially.
Usually, one uses RL instead of supervised learning because one has no choice, not as a matter of speed. As far as I know, it is everywhere and always the worse option. RL comprises (more or less by definiton) a set of methods for estimating gradients from data, via experience, which one must do when one is not provided with the correct targets for learning. In the case of trackmania, these would be the optimal actions given any state, which are unknown.
I think you were trying to point out the sparse reward vs dense reward issue. The "naive" reward structure of the task in trackmania is completion time, and that is a reward you see rarely relative to action selection (sparse). Including proxies for performance can make the reward structure denser and improve learning speed dramatically.
Edit: I now see that maybe you meant RL is a fast optimizer relative to something like an evolutionary method. Fair enough. This just tripped a wire for me because RL has very bad sample complexity, generally speaking.
We must keep in mind what computers are, and what humans are. They are different.
Computers are syntactic engines. They do syntax, and syntax only. They are incapable of "understanding" anything. They are incapable of doing semantics.
Humans, on the other hand, really "understand". Humans do semantics.
This is a major difference in capabilities, which will never be bridged, due to the nature of what creatures (we created them) computers are, and what creatures (God created us) humans are.
In addition, there *are* in fact fundamental limits, akin to the Shockley–Queisser limit, for AI. Gödel's incompleteness theorems, for example. Gödel destroyed Hilbert's dream of a completely axiomatized mathematics. This has major implications for the limitations of the capabilities of AI.
We live in an age where generative AI is making very impressive advancements, but it is important to keep in mind that GENERAL AI is not making such advancements. General AI is stagnant. According to Gödel's demolishing of Hilbert's dreams, General AI will never make much progress.
This is, partly, due to the difference between what kind of creatures computers are, and what kind of creatures humans are.
It should be self-evident to anybody who understands how computers work that they are syntax engines, and only syntax engines. It matters what symbols come in, in what order, and from what interfaces (keyboard, mouse, network, etc.) But they are all just (to the computer) meaningless symbols.
This symbolic manipulation is exactly what David Hilbert had in mind with his ambitious mathematical goals last century.
It should also be self-evident to any human who can understand Gödel's incompleteness theorems that humans really DO understand what Gödel put forth. Humans are doing semantics in that case.
So humans have a capability that computers do not have, and can NEVER have, if Gödel is correct.
And nobody doubts Gödel!
I have ordered the book, and look forward to reading it.