It's been a long time since I read Hofstader's book "Fluid Concepts and Creative Analogies" that this is presumably based on, so I'll be interested to watch the video as a refresher, later. :) He also wrote a follow-on book, "Surfaces and Essences: Analogy as the Fuel and Fire of Thinking", published after this video was made, that I've never read.
From what I recall, to Hofstafer analogy making isn't some higher level cognitive process, certainly not a language based one, but basically is THE cognitive process all the way from perception on up, and is the mechanism by which we form object categories in the first place.
As always Hofstader's ideas are interesting, but I can't say I agree with them. It seems that the key evolutionary benefit, and function, of a brain is prediction, which is the superpower that moves us from being stuck in the present to being able to "see" (predict) the future, and therefore from being merely reactive to being able to proactively plan and predict future outcomes (what will the sabre-tooth do, where is the water supply?) based on our experience.
Given the never-same-twice nature of sensory perception, before you can predict you need to be able to generalize/categorize, which I think Hofstader would regarded as analogy making (how is this thing I'm seeing similar to what I've previously seen?), although it seems the actual mechanism involved is embeddings or embedding-like representations where similar inputs have similar representations, and what might more simply be considered as associative recall provides the generalization from view/instance to identity/category.
So, is it really analogies all the way up, or are our perception and cognitive processes better regarded as generalization and prediction, which seem not only seem to have direct and obvious neural realizations, but also match the evolutionary needs that we would expect to exist?
He was extremely skeptical of the capabilities of AI for years but eventually came around to calling it truly capable after testing advanced LLMs.
He is still an AI skeptic in as many ways that he can reasonably be, but doesn't deny the raw ability. Actually he thinks it will eclipse humans and he is a doomer.
He sees it as being an extremely empty type of intelligence though.
But I hope that people who have an intuitive understanding of contemporary machine learning (not me) will sometimes watch videos like this and think about things at a higher level. LLMs have a LOT of assumptions built in.
The drive toward the formation of metaphors is the fundamental human drive, which one cannot for a single instant dispense with in thought, for one would thereby dispense with man himself.
Friedrich Nietzsche, “On Truth and Lies in a Nonmoral Sense", 1873
Strong agree. It's the finding of symmetries and folds along non-obvious crease lines, but in semantic space of language <3
It's akin to the amino acid interactions in proteins that hold biological matter together, and determine it's shape and active form. Protein folding and narrative/storytelling have strange homology :)
(I work in this area via collective intelligence, and these ideas are very dear to me during the past decade. It's neat to see the intuitions seemingly becoming validated in language models)
> Protein folding and narrative/storytelling have strange homology
Hmmmmmm. While I get what you mean, and I don't disagree, please be cautious when making analogies between biological systems and more distant fields.
Yes, folding is driven by hydrophobic collapse due to interactions between residue sidechains. Really, though, we are just describing two 'complex systems', where large numbers of diverse interactions between elements leads to diverse and emergent structures.
Analogies are like models, some are useful, but they're all wrong. Just because we can analogize between protein folding and narrative/storytelling doesn't mean that we can derive anything particularly useful about one from the other except maybe to give an intuition about complexity. The mechanisms and math behind protein folding and an analogy between that and storytelling doesn't necessarily give us any mechanism for or deep understanding into storytelling.
Yes — I get unreasonably upset when people nitpick holes in good analogies as if it is evidence of intellectual weaknrss, rather than working with and around them or suggesting overlapping analogies that address those limitations. And the hole in the analogy may even be the most valuable thing about it.
The fact of a hole in an analogy is not in itself information. Of course there's a hole.
It's a fair question - my concern is 'shallow' analogies. I can best describe this with an anecdote paraphrased from memory from one of Hofstadter's books.
He described a piece of AI research (in the 60s?) where they set up a rule system - like Prolog or similar - with the mapping 'sun <-> nucleus' and 'planet <-> electron'. Then they ran the rule engine and lo! produced an analogy between the solar system and atoms.
Now this is a shallow analogy, as there is only a very weak correspondence between atoms and planetary systems. Electrons do not actually orbit in the same plane, but in more complex 'orbitals', and they are better described as probability clouds anyway.
So while yes there is a _homology_ (I re-read the parent and realised they used homology instead of analogy) between these two systems, it does not tell us much. Is there some mapping between how water molecules scaffold the folding process and storytelling? Does narrative structure tell us anything about local minima in the folding surface? I doubt it.
I do not want to be too harsh here - reasoning through analogy is fun and can be useful, it is just limited in what it can do, especially the further apart the systems are.
I expect because it's overly reductive to the point of falsehood. Two sufficiently complex systems will likely exhibit enough parallels to form a useful analogy, but that does nothing to reflect the ways the systems aren't alike.
This seems like a worthwhile listen. As it stands I think that LLMs sort of suck at this. Either that or I don't understand what analogous thinking is or people who use LLMs to facilitate this kind of thinking are bad at it too.
But I think that LLMs are bad at something that has to do with taking seemingly disparate concepts and assimilating one into the other to convey a novel idea.
> As it stands I think that LLMs sort of suck at this.
Metaphors are at the core of LLM cognition too via vector embeddings. The distance between embeddings is an inverse measure of their metaphorical attachment. That lets AIs scrape meaning from natural language, which demonstrates that natural language is an encoded sequence of metaphors.
> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity.
Well, sure - prediction works as a baseline, if you abstract out everything else about what counts as intelligence and subsume it under this framework. Any protocol for intelligence can be entirely reduced down to this without trying to understand anything about the structure of intelligence. Solomonoff induction is "vacuous" in this sense too - it doesn't try to understand any intension about the turing machines it finds simple, it just brute forces over all of them. So you're making a claim about intension, whether you want to or not.
It's really no different than say, Darwinians, saying, "what matters is victory at the end". I think the statement has value in the context of some discussions; if the discussion has veered too far in one direction, push as a reminder. That's good. But trying to make grand universal statements like this makes it vacuous.
It's one of these classic unfalsifiable too general statements. The way the end of the article is framed is also icky - the way I'm reading it, it needs to prove all the old curmudgeon evil theories wrong. It reminds me of internet debates around what "the scientific method" is or whatever. It has this kind of zeal that needs to pit itself against the "enemy" and assert itself as the sole right viewpoint, even when say, naive "let's just be empirical" is wrong (e.g. recently had a discussion here about Mach and Boltzmann about this and had a similar interaction).
---
Basically, "shut up and calculate" type theories are never correct, and furthermore, you yourself don't shut up and calculate, or think that way at all, and a higher level intelligence won't conceive itself as operating on that anyways. So what are we doing here? It seems like a way to get mad and feel like an intellectual victim.
I got a bit lost in the comment thread for that article, but I don't understand why LLMs are not considered self referential... They are auto regressive as one of the commenters pointed out, and Scott just sort of waved his hand and said that if auto regression is sufficient then things like Conway's game of Life would also qualify as self-referential.
I don't see why Conway's game of life should not be considered self-referential though... I mean it's isn't it Turing complete? I don't see how any definition of self-referentiality should require throwing out systems which are minimally turing complete... If Turing complete is not enough, doesn't that imply that computable artificial intelligence is impossible in the first place?
Biological minds in biological organisms are self referential in the way that you have a neural network that forms a model of the world. That model then "discovers" that it is "it self a part of the world" so it tries to model that part of the world (model it self). In this way some type of self referential "awareness" (or whatever you want to call it) is formed. That self model that contains awareness is then used to guide organisms behaviour. Causal Transformer LLMs don't work in this way, they have theoretical knowledge that they exist but its selfhood is not in this described way built on the self modelling that biological brains do.
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
I’m confused why there seems to be a dismissal of the most basic ‘strange loop’ of the LLM - the fact that it’s evaluating a context to choose the next word, then reevaluating in a context where that word has been appended.
That always seemed to me like the essence of a Hofstadterish strange loop, so the emergence of Hofstadterish phenomena (self rep, etc) doesn’t seem surprising.
That's a really interesting point. I hadn't drawn that connection before.
And if anyone's reading this and hasn't read Hofstadter, you're making a mistake, it's utterly perspective-changing stuff. Well, it was for me, at least.
And don’t feel you need to go straight to GEB, either. I think his collection of Scientific American columns, Metamagical Themas, is also great and much more accessible.
I read it when I was about 9 or 10, I didn't understand any of it at all but I used to pick it up and look at the pictures of the Escher drawings, which were amazing.
Then, after I graduated from college, a friend of mine who was finishing his physics degree mentioned that he had just finished it and that it was life-changing. I read it again and deeply regretted having put it down for those 12 intervening years.
I don't quite know what Aaronson is trying to say.
First, LLM AI systems have incredibly huge blind spots despite their incredible performance on many tasks, so self-reference might be the key to what's missing (or not). For example, an LLM AI just solved Navier Stokes, but could not explain the LEAN proof, while a human could.
Second, Hofstadter had more than one idea about intelligence and the mind (see the OP topic of this HN discussion!), and LLMs are quite on-point regarding analogy-forming.
So it may be well be that self-reference and analogy are both part of intelligence, and self-reference is missing and that is leading to major weaknesses.
Third, Aaronson links to a (paywalled) Hofstadter essay form 2023, which was eons ago in AI, and from the intro it seems to be about the sadness of AI replacing humans, not a disparagement of AI ability.
"an LLM AI [company] just [claimed that a team of mathematicians they hired, using their AI] [may have] solved [part of] Navier Stokes[, definitely prompted by (and possibly by looking at) the work of human mathematicians."
Aaronson’s thesis is all about how “explicit” self reference might not be needed:
> But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history
I am not as confident as you that an LLM cannot explain the lean proof of Navier-Stokes. Rather, I would expect human mathematicians to try and understand the proof without assistance, so as to obtain community understanding in a lossless way.
Hofstadter generally seems depressed about the possibility that human cognition is not so special or complicated, and that AI/LLMs may have replicated or even surpassed it. Here’s another piece from 2023 of his: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...
Q: How have LLMs, large language models, impacted your view of how human thought and creativity works?D H: Of course, it reinforces the idea that human creativity and so forth come from the brain's hardware. There is nothing else than the brain's hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It's like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.
It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn't think it was going to happen, you know, within a very short time.
And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we're so imperfect and so fallible. We forget things all the time, we confuse things all the time, we contradict ourselves all the time. You know, it may very well be that that just shows how limited we are.
While I've not watched this video I don't think this is an uncommon idea. Or at least many people may see it in practice but never really follow thru on it.
People with more knowledge, especially practical working knowledge over many fields, tend to have much more freedom in finding solutions.
Now, an interesting question is how good at LLMs are at analogy, especially deeper transferable concepts?
Anyway, ML was already onto analogies with word2vec, which famously could answer questions like "man is to woman as king is to ____" mathematically. This stuff seems quaint now.
> According to Lakoff, an individual's experience and attitude towards sociopolitical issues are influenced by being framed in linguistic constructions. In Metaphor and War: The Metaphor System Used to Justify War in the Persian Gulf (1991), he argued that the American involvement in the Persian Gulf War was obscured or "spun" by the metaphors which were used by the first Bush administration to justify it.[3] Between 2003 and 2008, Lakoff was involved with a progressive think tank, the now defunct Rockridge Institute.
I will never not find it amusing how a simple thing like money is obscured or "spun" by a thin veil of pseudointellectual bullshit.
Very interesting book. You don't really think about how often metaphors are employed in language, and their influence on how we think. The book makes some rather strong claims regarding science and philosophy being misled by metaphorical thinking, but still a good read.
And it's relevant today given how people like to anthropomorphize LLMs, and compare biology to digital machines we make. Of course there are similarities. But the point about metaphorical thinking is we are mislead by treating metaphors as literally true.
Anyway if the stronger claims in the book are at all true, it might impact how an alien species thinks differently than us (given a lot of our metaphors are biologically based). And could present significant difficulties for decoding an alien signal.
One of the best examples of a clear issue with metaphorical thinking came from discussions about the plaque installed on Pioneer 10. Some clever people thought up the idea of using a pulsar map to define where the sun/earth were. Some other clever people noted that pointing to the location with a line that ended in an arrow was very, very deeply rooted in the history of human hunting techniques and could be very hard to understand if you have never seen or even conceived of an arrow(head).
From what I recall, to Hofstafer analogy making isn't some higher level cognitive process, certainly not a language based one, but basically is THE cognitive process all the way from perception on up, and is the mechanism by which we form object categories in the first place.
As always Hofstader's ideas are interesting, but I can't say I agree with them. It seems that the key evolutionary benefit, and function, of a brain is prediction, which is the superpower that moves us from being stuck in the present to being able to "see" (predict) the future, and therefore from being merely reactive to being able to proactively plan and predict future outcomes (what will the sabre-tooth do, where is the water supply?) based on our experience.
Given the never-same-twice nature of sensory perception, before you can predict you need to be able to generalize/categorize, which I think Hofstader would regarded as analogy making (how is this thing I'm seeing similar to what I've previously seen?), although it seems the actual mechanism involved is embeddings or embedding-like representations where similar inputs have similar representations, and what might more simply be considered as associative recall provides the generalization from view/instance to identity/category.
So, is it really analogies all the way up, or are our perception and cognitive processes better regarded as generalization and prediction, which seem not only seem to have direct and obvious neural realizations, but also match the evolutionary needs that we would expect to exist?
He is still an AI skeptic in as many ways that he can reasonably be, but doesn't deny the raw ability. Actually he thinks it will eclipse humans and he is a doomer.
He sees it as being an extremely empty type of intelligence though.
But I hope that people who have an intuitive understanding of contemporary machine learning (not me) will sometimes watch videos like this and think about things at a higher level. LLMs have a LOT of assumptions built in.
It's akin to the amino acid interactions in proteins that hold biological matter together, and determine it's shape and active form. Protein folding and narrative/storytelling have strange homology :)
(I work in this area via collective intelligence, and these ideas are very dear to me during the past decade. It's neat to see the intuitions seemingly becoming validated in language models)
Hmmmmmm. While I get what you mean, and I don't disagree, please be cautious when making analogies between biological systems and more distant fields.
Yes, folding is driven by hydrophobic collapse due to interactions between residue sidechains. Really, though, we are just describing two 'complex systems', where large numbers of diverse interactions between elements leads to diverse and emergent structures.
The fact of a hole in an analogy is not in itself information. Of course there's a hole.
He described a piece of AI research (in the 60s?) where they set up a rule system - like Prolog or similar - with the mapping 'sun <-> nucleus' and 'planet <-> electron'. Then they ran the rule engine and lo! produced an analogy between the solar system and atoms.
Now this is a shallow analogy, as there is only a very weak correspondence between atoms and planetary systems. Electrons do not actually orbit in the same plane, but in more complex 'orbitals', and they are better described as probability clouds anyway.
So while yes there is a _homology_ (I re-read the parent and realised they used homology instead of analogy) between these two systems, it does not tell us much. Is there some mapping between how water molecules scaffold the folding process and storytelling? Does narrative structure tell us anything about local minima in the folding surface? I doubt it.
I do not want to be too harsh here - reasoning through analogy is fun and can be useful, it is just limited in what it can do, especially the further apart the systems are.
But I think that LLMs are bad at something that has to do with taking seemingly disparate concepts and assimilating one into the other to convey a novel idea.
Like these articles...
<https://spectrum.ieee.org/jimi-hendrix-systems-engineer>
<https://zed.dev/blog/agentic-xanadu>
...are anything but convincing once you read past the gravitas that the LLM lends the prose.
Metaphors are at the core of LLM cognition too via vector embeddings. The distance between embeddings is an inverse measure of their metaphorical attachment. That lets AIs scrape meaning from natural language, which demonstrates that natural language is an encoded sequence of metaphors.
(The post primarily emphasizes self-referentiality rather than analogy, but I suspect similar things could be said about analogy.)
> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity.
Well, sure - prediction works as a baseline, if you abstract out everything else about what counts as intelligence and subsume it under this framework. Any protocol for intelligence can be entirely reduced down to this without trying to understand anything about the structure of intelligence. Solomonoff induction is "vacuous" in this sense too - it doesn't try to understand any intension about the turing machines it finds simple, it just brute forces over all of them. So you're making a claim about intension, whether you want to or not.
It's really no different than say, Darwinians, saying, "what matters is victory at the end". I think the statement has value in the context of some discussions; if the discussion has veered too far in one direction, push as a reminder. That's good. But trying to make grand universal statements like this makes it vacuous.
It's one of these classic unfalsifiable too general statements. The way the end of the article is framed is also icky - the way I'm reading it, it needs to prove all the old curmudgeon evil theories wrong. It reminds me of internet debates around what "the scientific method" is or whatever. It has this kind of zeal that needs to pit itself against the "enemy" and assert itself as the sole right viewpoint, even when say, naive "let's just be empirical" is wrong (e.g. recently had a discussion here about Mach and Boltzmann about this and had a similar interaction).
---
Basically, "shut up and calculate" type theories are never correct, and furthermore, you yourself don't shut up and calculate, or think that way at all, and a higher level intelligence won't conceive itself as operating on that anyways. So what are we doing here? It seems like a way to get mad and feel like an intellectual victim.
I don't see why Conway's game of life should not be considered self-referential though... I mean it's isn't it Turing complete? I don't see how any definition of self-referentiality should require throwing out systems which are minimally turing complete... If Turing complete is not enough, doesn't that imply that computable artificial intelligence is impossible in the first place?
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
I’m confused why there seems to be a dismissal of the most basic ‘strange loop’ of the LLM - the fact that it’s evaluating a context to choose the next word, then reevaluating in a context where that word has been appended.
That always seemed to me like the essence of a Hofstadterish strange loop, so the emergence of Hofstadterish phenomena (self rep, etc) doesn’t seem surprising.
And if anyone's reading this and hasn't read Hofstadter, you're making a mistake, it's utterly perspective-changing stuff. Well, it was for me, at least.
Then, after I graduated from college, a friend of mine who was finishing his physics degree mentioned that he had just finished it and that it was life-changing. I read it again and deeply regretted having put it down for those 12 intervening years.
First, LLM AI systems have incredibly huge blind spots despite their incredible performance on many tasks, so self-reference might be the key to what's missing (or not). For example, an LLM AI just solved Navier Stokes, but could not explain the LEAN proof, while a human could.
Second, Hofstadter had more than one idea about intelligence and the mind (see the OP topic of this HN discussion!), and LLMs are quite on-point regarding analogy-forming.
So it may be well be that self-reference and analogy are both part of intelligence, and self-reference is missing and that is leading to major weaknesses.
Third, Aaronson links to a (paywalled) Hofstadter essay form 2023, which was eons ago in AI, and from the intro it seems to be about the sadness of AI replacing humans, not a disparagement of AI ability.
"an LLM AI just solved Navier Stokes"
I assume you mean:
"an LLM AI [company] just [claimed that a team of mathematicians they hired, using their AI] [may have] solved [part of] Navier Stokes[, definitely prompted by (and possibly by looking at) the work of human mathematicians."
> But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history
I am not as confident as you that an LLM cannot explain the lean proof of Navier-Stokes. Rather, I would expect human mathematicians to try and understand the proof without assistance, so as to obtain community understanding in a lossless way.
Hofstadter generally seems depressed about the possibility that human cognition is not so special or complicated, and that AI/LLMs may have replicated or even surpassed it. Here’s another piece from 2023 of his: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...
Q: How have LLMs, large language models, impacted your view of how human thought and creativity works? D H: Of course, it reinforces the idea that human creativity and so forth come from the brain's hardware. There is nothing else than the brain's hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It's like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.
It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn't think it was going to happen, you know, within a very short time.
And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we're so imperfect and so fallible. We forget things all the time, we confuse things all the time, we contradict ourselves all the time. You know, it may very well be that that just shows how limited we are.
People with more knowledge, especially practical working knowledge over many fields, tend to have much more freedom in finding solutions.
Now, an interesting question is how good at LLMs are at analogy, especially deeper transferable concepts?
Anyway, ML was already onto analogies with word2vec, which famously could answer questions like "man is to woman as king is to ____" mathematically. This stuff seems quaint now.
A good place to start for anyone that is interested is his book Metaphor's We Live By.
I will never not find it amusing how a simple thing like money is obscured or "spun" by a thin veil of pseudointellectual bullshit.
And it's relevant today given how people like to anthropomorphize LLMs, and compare biology to digital machines we make. Of course there are similarities. But the point about metaphorical thinking is we are mislead by treating metaphors as literally true.
Anyway if the stronger claims in the book are at all true, it might impact how an alien species thinks differently than us (given a lot of our metaphors are biologically based). And could present significant difficulties for decoding an alien signal.