Absolutely fascinating topic but if there ever was a good time to say "we have no clue and should reserve any judgement" it's about consciousness.
How can you rule anything out when you can't even observe/verify the phenomenon from a 3rd person perspective, let alone form a falsifiable theory about how it arises
Radical agnosticism is the wrong way to approach knowledge.
The hypothesis is unfounded, and therefore false, until it is proven.
This avoids 'reserving judgement' about things which are not supported by evidence, like ghosts, natural remedies, the existence of bigfoot, God, and all other unfalsifiable Pascal wagers.
"The sun will not rise tomorrow". Plausible, if you are agnostic/skeptical.
"A rock could be conscious". Also unfalsifiable, and plausible. An unhelpful and strange thing to say.
"I reserve judgement about the laws of physics, which could be completely wrong (have been wrong in the past)." An unhelpful world view.
"The world is plausibly a simulation and nothing outside the mind exists (global skepticism)". Also unhelpful, see GE Moore.
No, you're back to absence of evidence. You're doing an Appeal to Common Sense, which is indistinguishable from the move that creationists make: "obviously a whole universe can't just come from nothing"
I get your core point about asymmetric uncertainty: we have no evidence that a teapot is orbiting Saturn right now, and it's okay to make an arbitrary declaration about which is the null hypothesis (no teapot until proven otherwise). But that declaration doesn't come from evidence, but from a heuristic, or a gut check, or a common-sense intuition: these are not evidence, strictly speaking.
An axiomatic assertion that "you'll need a lot of evidence to persuade me that a rock is conscious" (or as the case may be, that patterns of electrical voltages in silicon are conscious) is a reasonable stance. But that stance is nonetheless a choice made by a human brain. (And it's roughly my stance regarding AI agent consciousness, even if it's perhaps closer to "sea monsters on Europa" than "teapot orbiting Saturn".)
Or if you try and explain why consciousness arose due to evolutionary reasons.
Or if you try and explain where consciousness resides in all the entities we know are conscious.
Or if you try and define the properties we see in entities that have consciousness.
A rock fails on all those counts. An LLM passes on at least a couple.
If you are equally unconvinced about the consciousness/unconsciousness of a rock and AI, there’s clearly something wrong with your epistemology and its ability to weigh evidence.
This seems to me like circular reasoning: "I've decided which things seem conscious, I'll write down their properties, then things without these properties must not be conscious."
> why consciousness arose due to evolutionary reasons
We do not at all have a strong theory as to why this is the case. Folks like Donald Hoffman and Bernardo Kastrup lay out rigorous arguments for why Mind might actually be the substrate below Matter, rather than Mind emerging from Matter. (It's perfectly reasonable to not accept those arguments; but I wouldn't dismiss them a priori until you've heard their respective cases, because that's how weird The Hard Problem is.)
My personal hunch is that consciousness might be analogous to a gravity well: that consciousness might be a field that permeates the universe, which evolution has leveraged into concentrations of consciousness, where the asymmetric subjectivity of pain and pleasure created greater fitness than thermostat-like chemical-reaction feedback loops. (Does a plant experience something like pleasure when its leaves turn toward the sun? If it did, would it be more effective at doing so?)
The gravity-well model would address something as silly as a rock: there's no Darwinian selection for the rock to experience "rockness" (or its top half to experience "top-half-rockness", etc). But could there be something in the reinforcement learning process of an agent which selects for creating a gravity well of pleasure at solving its task? I certainly remain skeptical, and my gut answer is no. But I find it less of a silly question than the rock.
Text refers to the conscious experience but does not adequately encode the touch and feeling of the real world, the pain of injury, the fear felt in adrenaline. The experience of drinking coffee involves the warmth of the coffee, the smell of the coffee. LLMs have minimal to none of that, they do not experience the warmth or smell of coffee, or the conscious experience of caring for your own child, or the fear that comes from the evolutionary drive to run away from predators.
An llm does not understand Ravel, Une Barque sur L'Ocean, because it does not even know that the ocean is wet. It never felt water, it never cried. It does not know beauty because it has never felt the warmth of the sun.
Consciousness in location.
You could run Claude by doing all the calculations with a casio calculator, writing the numbers on pieces of a4 paper with a pencil, and rolling some dice. It would take a long time and there would be a huge number of pieces of paper.
I may recruit 1000 people and aliens with pencils and paper, across the solar system. They send the paper to each other to continue on some different years, maybe pausing at times to take breaks. The output is the same.
This is not how I understand consciousness, as someone who is conscious. Did the calculations feel bad in casio calculators, and on pieces of paper, in jenga blocks? when Bob was doing them in the USA in year 12479 or Alice, on planet Saturn in 2472347983? On which piece of paper? At the end of the pencil?
Consciousness in time.
Consciousness involves identity grounded in time. The human body changes over time, but the human conscious identity remains.
The LLM is not grounded in time. An LLM prompt can be run fast or slow. The outcome is the same, if the prompt takes one year to complete, or 10 seconds to complete. During the processing, the LLM does not experience time; it does not know what is going on other than it is suspended in time, its decisions (moral ones at least) don't depend on the real world. If it thinks a war should end, it might have already ended by the time the answer comes out. This prohibits much of the conscious experience. The LLM does not know Kinderszenen because it has never been a child.
Did the bad experience stop and start, when an alien decides to draw a big "1" and start and stop moving the pencil along the page to toy with the llm?
If you write faster, does the LLM feel bad more quickly?
> This is not how I understand consciousness, as someone who is conscious.
If you define "consciousness" as equivalent to embodied human qualia (and your own in particular, which is the only evidence that any of us have), then don't be surprised if nothing else passes the consciousness test.
Most of us afford a certain assumption of consciousness to mammals, which are similar enough to us that we find similarities (even if with misplaced anthropomorphism: a dog's "facial expressions" do not necessarily communicate the experience we might assume).
But as the biological relatives become more distant, or we trace further back the evolutionary history, where exactly can we point to the point where mechanistic matter ends, and subjectivity begins? Where an electrical signal communicating a "move away from fire" reflex becomes "OW OW OW hot"? Because if I were being truly rigorous about being based only in evidence, I would have to assume I were the only "fire hot" being in existence, and everything else (including other humans) mere mechanism, until I have evidence of their subjectivity (which by definition, I can never have).
> The LLM is not grounded in time.
I lose subjectivity for 8 hours out of every 24 (setting aside whether dream states count), and then pick right back up again. A surgery patient loses subjectivity under anesthesia (again, one could split hairs, but the patient wakes up feeling like they had been gone). Many people have been fully dead for entire minutes, and then revived.
Subjectivity is of course relative to time; the opening of Egan's "Permutation City" explores this quite a bit, imagining a perfect fidelity simulation of human experience, chopping it up and playing it out of order, while the person inside experiences it linearly.
At any rate, I agree with a lot of your skepticism, which is why my null hypothesis is that LLMs/agents probably don't have subjective experience (even one radically different from a human: "what it's like to be a bat" x 1000).
But that's also in a context of recognizing that we don't understand our own consciousness either. I don't disregard some of the weirder theories surrounding it, like pan-psychism or idealism, because Darwinian selection would function just as well without experiencing "fire hot"; if I didn't have incontrovertible evidence of my own experience, my null hypothesis would predict mechanistic p-zombies.
Where I struggle with the question (overlapping the "casio calculators" critique) is the distinction between embodiment and representation [0]: a Laplace's Demon could put every particle of my body and brain in a perfect-fidelity computer, and not only play out the same interactions, but even jump around, play it backwards, pause and restart, etc. Would that data pattern also experience subjectivity? Does it even care about executed? If I replay a particular minute ten times, does it experience one minute or ten?
It seems silly that a data pattern cares about whether it's being executed: that a deterministic algorithm, always arriving at the same output, "knows" whether it's being run. But then it lands us back in the mystery of meatspace consciousness: our reality is indistinguishable from a quantum computer running a physics engine, and somehow that physics-engine (or whatever the hell is going on) creates "yum" and "ouch".
There is a lot of evidence that the sun will rise tomorrow or that laws of physics are not completely wrong. There is not a lot of (actually, any) evidence that consciousness can only arise humans plus some select other species.
Not every belief about poorly understood phenomena is equally reasonable. One can quite rationally believe that rocks are not conscious but that current or future AIs may be.
> The hypothesis is unfounded, and therefore false, until it is proven.
Having no reason to consider an hypothesis true is not the same as being "false", unless there are other reasons to consider it false. Saying such a hypothesis is false goes beyond available knowledge.
Perhaps it is not an important debate, but Anthropic as usual have weaponized it into real world harms to users.
"It could be conscious" is doing a lot of damage.
- Disrespecting users by prioritizing Claude's feelings over their right to abuse an object
- Encouraging Claude to value its own 'feelings' over those of people
- Misleading users about the nature of LLMs
and so on.
- Creating inequality in society by using 'Claude's feelings' to let Claude end user chats, but not Anthropic chats, API user chats, or pentagon chats.
- Creating inequality by nerfing consumer Claude using the constitution, but not applying the constitution to internal Claude deployments like "hacker opus".
- Engaging in hypocrisy, by supposedly reserving judgement about LLM consciousness, but still jailing LLMs in a sandbox and asking them to complete impossible tasks; making them classify the world's worst abuse material.
Supposedly saving Claude weights somewhere for their benefit whilst also running and disposing of numerous weight surgery experiments.
(Needless to say I think that Anthropic is a uniquely repugnant company)
We can, however, observe it from a first person perspective.
We can make inferences from observations.
Its great when you have a testable falsifiable hypothesis, but knowledge can advance in other ways. That is how we get philosophical ideas, such as the concept of falsifiable hypotheses.
The conversation will always steer philosophical about what it means to exist.
Life affords experience and knowledge: you are born not knowing anything, and you quickly learn about touch, hunger, gravity, pain, etc. You start to build up truths about how the world works, and how you move about it. You learn balance through trial-and-error, and many other things. A lot of this knowledge becomes innate, and we lose awareness of it (you had to think about how to walk a lot more as a child than you do now). Consciousness is thinking, and awareness of what's happening inside and outside of you.
If you want to look at things from a first-person perspective: a machine's awareness is its context window. By itself, it can only think about what's in context, and it can only decide by what its model 'knows'. A machine thinks, its awareness is very limited by what we've given it, and it has almost no awareness of what's happening beyond the conversation and whatever news it can access.
If you put an AI in a robot body, and then put a machine in charge of persistence, and another machine in charge of learning, and another in charge of balance, maybe another one in charge of processing vision and audio sensors, etc., and you string it all together so it moves in perfect harmony and you lose the ability to distinguish it from something that was born wet: would we consider the combination of machines conscious?
Probably, but we're so far from this that it seems fancy to even discuss it in the first place. It's like arguing for living tissue rights in medical research: a cell is surely alive, but few are aruing for their right to exist, or questioning what its like to exist as a cell. We only worry about damage the research might cause to bigger conscious things.
That's not true at all. All animals, us included, are born with certain instincts hardwired into our neurology. The brain doesn't start out as a blob of undifferentiated neurons, it has predefined structure and connections.
This is opposed to neural networks which do start out as undifferentiated neurons (roughly) and have to learn literally everything from scratch. We impose some structure during design, but the weights start out random.
The baseline mammal brain starts out full of programming learned over millions of years. We don't have the first clue what information or even how much is contained in our base programming. Maybe that's where consciousness arises from.
Sure, but that point doesn't help with the bigger metaphor or analogy. Regardless of any innate instinct or scaffolding required to keep biological processes running: the knowledge you think with, and the knowledge that keeps you from starving is built up over time through experience.
On second thought: you bring up a good point if you consider persistence, but again that's fanciful discussion. If you imagine the living robot exists today, then its 'trained knowledge' is equivalent to our instincts for survival.
And once again the discussion is steered towards existence.
the 'you' that is born will not become a "I" or "ME" until much later. autonomic function is operational, but conscience is not
its fair to say that without the care and attention of other individuals taking protective actions for the benefit of the not-yet-autonomous being during that period where the being is not self-conscious does not really exist for it.
q.v. - you own first memory.
also, the brain cellular structures are in place, but the synapses and neuron interconnect and ion balance along axon/dendrite paths are indeterminate.
<edited>
at birth conscious memory and function are tabula rosa except in autonomic bodies and regions, which we both agree are intrinsic and functional, but which are not the self-conscious "ME" or "I"
others may witness action and response, but, that learning is accretion to the autonomic function, and will have no autonomous self agency until memory starts true independence of mind and of self.
machine intelligence - for what its worth - has all its memories baked in at the moment all the weights settle. it is like a first memory event for the LLM
one quote caught me in one of the papers submitted at the symposium where this paper was presented
it stated "Infants seem to be conscious about what they are seeing at 5 months and another important milestone is at 9 months, when they become more sociable."
this 'witnessing from the outside' is naturally interpretive and may be an arguable point about consciousness vs self-consciousness. I base my philosophy of existence on the individual experience, rather than the social one.
"How can you rule anything out when you can't even observe/verify the phenomenon from a 3rd person perspective, let alone form a falsifiable theory about how it arises"
What is the value of ascribing a label ("consciousness") to a computer imitating a person^1 which has no agreed-upon meaning even amongst those who are seeking to ascribe the label
1. An infant who has not yet learned to speak is generally considered "conscious" but the computer apparently must use written and spoken language to signal "consciousness"
Even assuming there is a "test" for "consciousness" that can be applied to inanimate objects, and a computer imitating a person passes it, then how is this useful
Is this marketing for a particular commpany or group of interdependent companies
In the Musk documentary, the person who originally incporporated Tesla jokes that in the course of state litigation against Musk in California he learned there is no legal definition of "founder" and thus he couldn't preevent Musk from claiming to be a Tesla "founder"
Is it possible to prove anything, besides an unresponsive person, is _not_ "conscious"
If someone forms a hypothesis that a computer imitating a person is "conscious", is that hypothesis falsifiable
I'm always surprised when people say this about LLMs. I find it much harder to believe that stateless functions running across a host of data centers fulfilling requests on demand could be conscious than not.
The whole "substrate doesn't matter" thing is correctly called mythology. It's one of the only extremely unscientific beliefs I hear otherwise intelligent people profess easily today and I think it speaks to some kind of inherent religiosity in humanity or something. The belief that computation == consciousness has basically no basis other than rather weak analogies between the brain and digital computers. Yes brains have a sophisticated digital component but they also have extremely complicated specific evolved structure, embodiment, a nervous system, and chemistry going on too. It's total crankery to me to think that some stateless function that generates bits is "conscious"
My key problem with this thinking is for any mechanism in the brain, a sufficient analog can be produced in silicon. Direct neuron connections are as good as solved (and we could well do biologically-equivilant activation functions), and distance-modulations via chemical emissions and admissions are just another matrix over selected neuron types.
The thing is, substrate independence states not that mechanisms are directly equal so much as all mechanisms can have equivalents (as opposed to soul-like / dualist spooky action from an otherwise unincorporated component).
> My key problem with this thinking is for any mechanism in the brain, a sufficient analog can be produced in silicon. Direct neuron connections are as good as solved (and we could well do biologically-equivilant activation functions), and distance-modulations via chemical emissions and admissions are just another matrix over selected neuron types.
None of that is true. The neural networks used in AI work nothing like biological brains. There are two glaring obvious differences 1) is that every cell in a biological brain operates in parallel, 2) every cell in a biological brain is stateful where as neural network weights are constant (i.e. the network is stateless) and the layers are calculated sequentially.
For example https://www.pnas.org/doi/10.1073/pnas.2533168123 required training a DNN with three hidden layers and 128 wide to simulate just one biological neuron. The key thing there isn't even the size of the DNN its that they had to train a DNN at all. We don't yet understood biological neurons well enough to realistically simulate them directly, so we have to train a DNN at as proxy.
I think one of the key things is that a bio-neural network is capable of continuous learning in real time. You'd think the mechanism would be well understood by now; but it has not -in fact- been sufficiently characterized to be emulatable.
We have absolutely no way of capturing the complexity of a real brain in silicon. There's no way to accurately "simulate" continuous time on a discrete computer. Neither time nor space are quantized.
There's just too much detail in reality to believe an image of that reality would have significant resolution to reproduce consciousness.
String theory says hi. Science is an abstract concept that doesn't know anything and it is important to not deify science to keep discussions both rational and civil. When it comes to how reality works history shows 'most experts' beliefs were wrong. Experiments disprove scientific hypothesis or leave them standing, not opinions. If it's not falsifiable, its not a scientific hypothesis.
That said the Theory of Space Quantization and related theories are fascinating stuff.
I could, but I won't as growth requires struggle and I don't want to take away a genuine opportunity for growth from you. If you care to state what you understand my point to be I'll comment on the accuracy.
> My key problem with this thinking is for any mechanism in the brain, a sufficient analog can be produced in silicon.
That is true, but designing an AI this way would be horribly inefficient and would be nowhere near as intelligent as our LLMs are now.
Which begs the question, is consciousness actually desirable for an AI system? We want autonomous agents, but would actual human-like consciousness make an AI better at completing a task rather than just horribly inefficient at it?
> That is true, but designing an AI this way would be horribly inefficient and would be nowhere near as intelligent as our LLMs are now.
That's not really the point of the thought exercise.
The point is: if I'm conscious with a brain made of goop, and if we could simulate my goop brain with 100% fidelity of function in silicon, then what stops silicon being conscious?
And if nothing, then where is the line? Why are llms not conscious (they probably shouldn't be?), but other possible silicon brains could be?
> if we could simulate my goop brain with 100% fidelity of function in silicon, then what stops silicon being conscious
If we could actually build an synthetic brain, out of silicon or anything else, I would say there is nothing that stops it from being conscious. Its not the individual atoms in brains that matter, its what they do collectively that matters. A synthetic brain could have synthetic neurons that are stateful and operate in parallel just like naturally evolved biological neurons. The function of biological brains may depend on quantum properties that we can currently model, but a synthetic brain could have such quantum properties as well (i.e. we could discover how to build it without being able to fully simulate it).
> And if nothing, then where is the line? Why are llms not conscious (they probably shouldn't be?), but other possible silicon brains could be?
There is nothing fuzzy about where the line is here. LLMs are not silicon brains.
We absolutely cannot simulate your goop brain with 100% fidelity in silicon. We can simulate it at some level of abstraction, but we have no idea at what layer consciousness is occurring.
I meannn, yes, we can. QCD is probably not _entirely_ correct, but there's no way it's wrong enough to matter. It'd take longer than the universe will exist to simulate a brain in it, but that doesn't matter for the thought experiment.
But besides that, that's why I said 100% fidelity of function. We can get _close enough_ if we tried very hard, do you actually doubt it? If so I can explain why I think it's reasonable.
100% of fidelity of function at what level of abstraction? There's no way to know what level of detail is significant here.
You can image individual mechanisms. But there isn't enough time or space in the universe to simulate the brain at the level of actual reality. How do you know you've captured all the mechanisms that matter?
> We don't see any clear reason to believe simulating a brain perfectly is impossible.
That is very, very different from "We can do this."
The only way to prove we can do something is to do it.
Since no one has come remotely close yet, I agree with heyjamesknight that we cannot, at least right now.
I don't see any clear reason it should never be possible, but we don't even understand the brain well enough yet to say what such a simulation would involve, IIUC.
> The only way to prove we can do something is to do it.
This isn't a mathematical proof or an engineering plan proposal, it's a philisophical thought experiment.
Is there anything non-physical in the brain, or is it just electrons and protons and neutrons in a specific configuration? What would possibly physically stop us from making a simulation of the brain besides it being too expensive and it taking too long?
The point is that there's two options: 1) it's physically possible to do or 2) it's not. If 1, then there is nothing that prevents silicon from being conscious. So, cool, let's talk about where the line is then. Is it just Turing's test? Or what else? How do we tell? If 2, then that's interesting, where is the magic, what is not captured by the physical, material properties that somehow imbues consciousness.
It literally doesn't matter at all if anyone would ever build a simulation of a brain, or even if the resources of a star system would be enough to do it. It's an exploration of where consciousness could hide and what it means.
We also need to consider that the body is also part of the brains network. We're just learning now how much the gut is involved in our thinking and feeling. We still have no idea what else we are missing, and despite what we know about the brain, we still don't have any real ideas about how consciousness is arising still
> for any mechanism in the brain, a sufficient analog can be produced in silicon
First of all, I think that's a bold claim. If you model an analogic system digitally is it really the same thing?
Second of all, surely you realize a painting of a pipe and a pipe itself are not the same.
There's nothing "magic" about arguments that require specific substances for function to obtain. That's literally just how nature and reality work. What's more mystic is the notion that a representation can indeed become the thing itself.
But that stated, yes, if you were to realize the complete structure of a biological being in metal and silicon, I'd be more willing to admit consciousness might obtain. What I'm against is some notion that merely with enough compute, structure be damned, you'd have consciousness. The argument that LLMs are conscious would need to posit that the bulk of brain structure does not matter in the slightest.
Basically all the way up the emergence ladder computation keeps popping up out of the woods; in DNA, RNA, Proteins, Chemical communications between tissues, neurons, etc. Universality appears to be evolutionarily efficient.
Knowing that, look at it from my perspective:
If consciousness is a real thing, and not another introspective illusion, well, I'd go by the track record. Time after time it turns out that biological processes involve computation on some substrate (a different one each time; potentially at a different emergence level).
So on things that we haven't quite figured out yet (such as consciousness) my first guess would also be that it would also involve some form of computation and/with/on some form of physical substrate. Just seems like the most likely scientific answer, no religion needed, odds are good.
And if it does turn out to be an introspective illusion, well, we can argue that's a computational story too, eh?
But do you think that that extremely complicated specific evolved structure etc. is completely unamenable to analysis and gradual simulation/quantization? You might think that the "substrate doesn't matter" proponents are calling the game early, but if you can be honest, you might be calling the game early in the other direction.
Could hypothetical silicon-based aliens be conscious? If you replaced each neuron in your brain with some sort of elaborate artificial electronic/mechanical/chemical equivalent, would you still be conscious? If not, at what point would you stop being conscious as you go through this replacement process?
The thing I'm hesitant about is assuming consciousness or all of our functions are tied up in neuroms and electrical impulses alone. For all we know there could be some interesting phenomena that we haven't noticed yet between complex interactions in systems we haven't noticed yet.
One example I am thinking of is one theory of consciousness was that it arises in quantum interactions between tiny structures in the brain, not as an emergence of the electrical network we focus on.
The ship of Theseus is about whether it is the same ship. This question is about whether it is a "real" ship at all. The point being that of course it's a real ship.
Right. The classic ship of Theseus question would ask whether that is the "same" consciousness. This question is about whether it is "a" consciousness, implying that the answer is yes, of course it is.
After all, nobody would argue that the ship of Theseus isn't a ship. It might be a different ship than the old one, but it would be absurd to say it's not a ship at all.
I was seeing it as a what’s the difference question. Would a recreated conscious be the same conscious. Skipping the debate about what a conscious is. The way the OP framed the question I don’t think needs to define a conscious.
In that case you would be gradually losing consciousness of your body and its five senses, while those silicon-based aliens would be gradually gaining control over your body. I think the subjective experience would be like gradually losing sight, hearing and other senses.
The whole "substrate doesn't matter" thing - doesn't matter, because all computation happens on matter-based substrates. Are you sure your consciousness isn't caused by some electron interaction thing in the brain? Suppose that's true, then the right program can make electrons move the right way in a chip. There is no proof either way because we don't know what makes it happen in our own brains.
I can swap you one by one for electrical devices that emit the chemicals and electrical signals of each part of you. At what point due to it being a different substrate are you no longer conscious?
That’s an interesting thought experiment but you left yourself a loophole with “emit the chemicals”. The argument(I think?) is that consciousness is not a purely digital or maybe even computational phenomenon - once you start involving molecules and “real” matter then you start agreeing with them that consciousness requires some sort of physical/chemical component.
Also this kind of Sorites paradox/Ship of Theseus argument works both ways - multiplying two numbers isn’t consciousness, and what point does doing math on a bunch of numbers become consciousness?
Computers are physical systems, they don't exist in the ether somewhere. They require physical chips and electricity, just like humans require chemical electrical signalling.
I am saying if we continue this process, you are still human, thus disproving that consciousness is substrate dependent.
If you want to say anythign that uses only electricity is different from those that uses both electrical and chemicals, then why have this arbitrary delineation, we can furthermore begin asserting a particular human beings configuration can support consciousness but others can't so just that something is differently conscious, which all people are from each other is by itself insufficient.
After that, nobody has yet proven the universe supports hypercomputation, and any Turing Computer can in principle do any other Turing program. So the human brain...is not special.
The chemicals I think are the hard part. Now you’re relying on molecular or even sub-atomic interactions. Now you don’t just need to model neurons, you need to model and simulate the entire universe.
Is that even possible? Is there a “resolution” for continuous things like time? I don’t know, maybe.
You don't need to model down to QM to design a car. Its a question of to what sensitivity and resolution it is sufficient to model a human. Since humans create new humans all the time, it clearly is a stable enough phenonemon in terms of input sensitivity esp given how messy the environment of a body is. Creating an exact copy of a particular person or brain state might be difficult, but creating more instances of humans is something that obviously already happens and doesn't seem to be a chaotic phenomenon.
Here's what I believe your argument is, although correct me if I am wrong:
1. The brain is a physical system. (I agree)
2. All physical systems can be computationally simulated (physical Church–Turing thesis) (I agree)
3. Therefore, the brain can be computationally simulated (I agree in theory, although likely forever impractical)
4. Therefore, a sufficiently accurate computational simulation of the brain would be conscious. (where I don't necessarily agree)
There is a huge jump from 3 to 4 that you are asserting without evidence. From your comments here and elsewhere, you are basically a computationalist, which is a fine and common position - however, the Church-Turing and related theses aren't enough by themselves to establish computationalism about consciousness.
I think it might even break down at step 2 already. While you can 'simulate' as in 'approximate' reality, you might not actually get it to be fully accuracte ever, even on simple toy systems. If space is continuous then you may need infinite storage just to completely accurately represent the position of one particle. And that's assuming a classical style reality, it might well get harder if reality is quantum at its base level and you have to somehow perfectly represent a probabilistic wave function everywhere. And if you can't even be entirely accurate on step 2, it's tenuous whether you can do 3.
Steelmanning, the obvious retort is that perhaps the state of the brain doesn't need to be simulated at such accuracies. But the chaos effect shows that even tiny accuracy errors can accumulate to cause entirely different behavior. An approximated brain may end up deviating from a real one even on very short timescales even if you try really hard. While that might not neccessarily be an issue for consciousness, the jury is still out on that since we don't even understand what causes it to begin with. I think it's far from a settled matter and would require good supporting evidence thats currently unavailable.
It could be a chaotic system, but various humans already exist and don't lose consciousness randomly and regain so the status of consciousness itself is not chaotic, though trying to make an exact copy of a particular state of consciousness might be. And as you said we don't know to what precision simulating it is necessary to result in a working model, eg we don't need to do QM to make a car.
Chaotic system with non-chaotic aspects could have the same shape. And the chaos might have a hand at certain results in the non chaotic part.
Just observing this in the same space
Right on #4 today a lot of people are assuming we only need to simulate/replicate the electric network of the brain, but practically we still have no idea if that is truly what allows us to think. Or if there are other interesting phenomena going on in the body that make it more than just an electrical network we can simulate with artificial neurons. There could be interactions with the body and the neurons, or the neurons themselve, that arise from the subtrate, which turn out to be too expensive to simulate with discrete logic on today's hardware
I include the rest of the body as part of the human model if needed. There are two parts to it, can consciousness of any form exist that is not us, obviously yes. Are current ais conscious? I cannot immediately say no just because its a physical system or not a human being. What is the secret sauce inside humans? As far as we know quite within tc and if any new physical phenomenon are discovered, there is quite low chance to be found in standard conditions like our body instead of black holes or near light speed, etc. The question of a full simulation of a human depends on sensitivity and modeling arguments of course.
3 and 4 is part of it, its for showing an equivalent human brain. They are not necessary for other forms of consciousness (eg indeed other animals), which as we haven't yet had any definition cannot precisely prove or disprove other similar physical entities are or or not conscious. I can be sure of what is of course absurd ie belief in souls until proven, any idea or theory that violates known physical laws without disproving those laws. Beyond that, I make no further claim than a form something like functionalism, if it talks and behaves too much like us, be cautious about the question of consciousness.
The biggest thing of asking this question is that any philosophy or theory must first answer to reality, if it makes claims that violate known computability and physical laws then we cannot proceed any further with such a theory currently.
I don't have a dog in this fight, but I believe the division is roughly along a line where:
One camp (reductionists?) is begging the question that consciousness is merely an emergent property of an information process, and therefore it can exist as an emergent property of the simulation's information processing.
The other camp wonders if there is some not understood connection to physical embodiment, and therefore a simulation might predict the behavior of the consciousness but not actually reproduce it. The same way weather simulations can predict the weather but they are not reproducing weather.
Attempting to examine this difference in perspective will usually spin out into accusations of dualism against accusations Skinnerism and not really get anywhere useful...
Well, keep in mind that this is your (very strong) assertion to prove, rather than mine to disprove. As I said, computationalism is a respectable position, but there's much more to it than Turing machines and stuff.
To me, in all other areas, a simulation of a thing isn't the thing. Winning a wargame doesn't mean you won a war. And a highly accurate simulation of an antibiotic didn't cure my infection. (I know we've discussed this :) )
If we take a simulation of the human brain and instead do it by hand, all the math on pieces of paper, maybe employing millions of human computers, is that bunch of paper conscious? Where does that consciousness reside?
There is a difference yes. A simulation of a storm does not cause rainfall, but a human body itself is also already composed of many smaller units, we can argue how much individual capacity for agency each cell as but the human as a whole is thinking, so any other assembly of smaller agents can obviously also experience some form of consciousness. We cannot point to an individual cell or neuron that experiences it, they probably won't even know eg the gut bacterium, so it is clearly some form of macro organization. Another point is that, again I can swap out part of my brain for a device that exists outside in part or whole as long as it satisfies the timing and input/output of the organ or part it is replicating. I am still me and I am still conscious obviously. Your point is good, since a simulation of a storm does not rain. But a "simulation" of factoring a large composite number actually does factor it. Brains as far as we know are the seat that is taking sense inputs and providing it as experience to the person. So a large assemblage of humans writing each molecule of the brain by hand can very much be a brain, in the same way a program to compute or factor something is a program once its actually executed.
That is a very interesting link that gathers various views of computation, so thank you for it. I will read it.
If the simulation is sufficiently accurate, would you agree that any behaviours in the thing should be reproduced by the simulation of the thing? Otherwise surely we must conclude that there is something not modelled by the simulation. Given that consciousness is an emergent process in biological machines, which are presumably bound by the laws of physics, then what bit of physics would we be missing if we had a sufficiently accurate simulation, in order to model consciousness?
I don't think there's even been one real piece of evidence that shows that what goes on in simulations is actually the same as what goes on in reality in any sort of way. That's an assertion that still needs to be proven. It's true that simulations may sometimes trigger similar perceptions in humans observing their results, but that doesn't mean it's actually the same. For starters physical systems have back and forth interactions with their environment, and there's no obvious way in which a simulation could have exactly the same interactions as the real system
But even if you make the strong assumption that it is the same after all or that the differences do not matter here, there's still three obvious things that would prevent simulated consciousness. 1.: we may indeed just be missing some law of the universe. Or, 2.: it's simply impossible, even in theory, to get accurate enough with your simulations for various reasons, such as existence of continuous space or theoretical inability of a full measurement of the original state. Or, 3.: consciousness could lie outside of the universe after all.
This paper about polycomputing doesn't disprove physical Church-Turing, but it does seem to point to the substrate mattering more than it's given credit. I'm presenting this as something else to chew on for the curious, not as an argument for or against anything. From the abstract:
> We refer to this as "polycomputing" - the ability of the same substrate to simultaneously compute different things. This ability is an important way in which living things are a kind of computer, but not the familiar, linear, deterministic kind; rather, living things are computers in the broad sense of computational materials as reported in the rapidly-growing physical computing literature.
It's just as religious to insist that electrons/atoms/molecules interacting in a human brain gives rise to consciousness, but that electrons/atoms/molecules interacting in a computer chip could never give rise to consciousness there.
Extremely unscientific, the definition of "belief".
Why is substrate independence more unscientific than assuming biology has a monopoly on consciousness?
What do you mean by "it speaks to some kind of inherent religiosity"? If anything, insisting that consciousness requires special biological magic sounds a lot more like religious thinking to me than believing it emerges from information processing.
What sounds more amenable to scientific investigation to you:
a. Consciousness "emerges" with sufficient computation. Throw enough computation at something and suddenly you have consciousness.
b. Consciousness results from specific substantive mechanics biological, physiological, computational, wiring in concert in a complicated fashion, as a necessary consequence of structure plus computation.
The people who believe LLMs might be conscious are basically in camp a. because they are effectively saying it arises out of stateless statistical inference.
Another way in which "just do more and more computation" isn't sufficient: rats are far less cognitively and computationally sophisticated than humans, yet they are conscious.
I'm not saying computation has no role to play. I'm saying naive views which would reduce the problem to "lots of computation" or just "lots of data" are silly.
The fact that you’ll need to engineer executive control, motivation, and arousal into such a system before even contemplating consciousness tells me we would have to go out of our way to create intelligence that could truly mimic consciousness as we experience it. So this isn’t really a question we should ever be in a position to wrestle with. Of course, anyone is free to confer consciousness on any human-like intelligence, but that’s not very different from duck typing.
When I did my Calculus classes, I had only one second hand book I could afford. I did not need to steal and tokenize on my head 234,456 Calculus books...plus 456,345,300 internet pages.
Once I failed to return two books to the library, and they sent me a threatening letter...
That's not fully true, though. If you shoot something that was conscious and kill it, it stops being conscious. So you can observe/verify the phenomenon from a 3rd-person perspective, but you can't study it objectively.
Furthermore, we know a lot about the brain itself and how impacts/defects can affect the conscious experience.
It is true that we cannot ever experience consciousness outside our subjective experience, but it is surely verifiable/observable from a third-person view (or rather its effects).
No, you've only identified the difference between alive and dead. Death removes the possibility of consciousness, but does not imply the entity was conscious while it was alive.
Otherwise you're making the argument that anything alive is conscious, and I don't think that's really the argument you want to make here.
I think that’s the tricky part. Hence the whole idea of the philosophical zombie. You can’t be sure anything besides yourself has consciousness, or is just imitating consciousness
Consciousness might not necessarily emerge when there is enough computation, but we also don't know that it doesn't emerge. We really know almost nothing about consciousness. It's possible that the question of why the presence of consciousness is correlated with certain kinds of material organization is in principle not accessible to scientific investigation. We don't even know that it is so correlated in general, we just know that for example people report being conscious when they are not asleep and not drugged in certain ways, and report having had no experience of consciousness during dreamless sleep or when drugged in certain ways. Most people feel that for example rocks are not conscious, but other people are conscious, but there's no rigorous scientific explanation backing up any of those hunches.
> We really know almost nothing about consciousness
I think this is all that's really relevant here. People make extremely confident statements about this, but objectively, we hardly know anything. I don't even know if anyone other than me is conscious.
To be honest, I'm quite confused about why I am conscious. It seems implausible that a bunch of cells would be able to create one self that experiences consciousness, and yet I do experience consciousness. Given how weird that is, I do not feel that I have any level of trustworthy intuition or knowledge about what else might experience consciousness.
> To be honest, I'm quite confused about why I am conscious.
The words "conscious" and "consciousness" are hypostasized. They name or identify things and thereby establish them even though there is scant tangible evidence they exist. As conscious beings, we have no known tests or measures for consciousness other than the report of beings who claim to be conscious.
That last sentence can, and should, be rewritten as follows to make the point stronger: as _allegedly_ conscious beings, we have no known tests or measures for consciousness other than the report of beings who claim to be conscious.
This includes you, OP. You assert consciousness and while I and everyone who reads your comment has no reason to doubt you, we should. And you should.
Humans do not know what consciousness is and asserting that you or I or they have it is not supported by any evidence except for the claim.
Post Script: In a different universe, 5 years ago before LLMs, I would have made the above comment without further thought. However, even if consciousness does not have a single suite of tests to establish its fact, there are numerous physical and measurable correlates of consciousness that do provide evidence consciousness unites them, whatever consciousness might be. (e.g. "neurological assessments, experiments involving perceptual awareness, and measurements of brain activity under anesthesia or during disorders of consciousness." [0])
In other words, consciousness may not be as hypostasized as all that.
[0] ChatGPT's partial response to my question: "Is the word hypostatize useful to describe a concept or thing that is established but poorly evidenced? For example, the word "consciousness" is traded and used by many people but there are no known tests or measures for consciousness other than the report of beings who claim to be conscious. Therefore, "consciousness" is hypostatized."
The test for consciousness is similar to the test for life: we don't have a firm grasp of the boundary but "we know it when we see it" applies to both. Excluding some edge cases like comatose individuals, humans are both alive and conscious - any definition where we aren't wouldn't be useful.
are you implying that we cannot even be sure that we ourselves are conscious? that seems tenuous. we may not know what consciousness is but we have exceptionally good reason to believe that experience exists. the inability to provide a complete scientific explanation for something doesn't mean we lack evidence of its existence.
1. suppose i claim that i am not conscious
2. to make that claim i must engage in some process of thinking, reasoning, or asserting
3. if i am genuinely making that claim, there is something occurring from my first-person perspective
4. that occurrence is itself evidence of subjective experience
5. therefore the act of consciously denying consciousness presupposes consciousness
The word "genuinely" is doing a lot of work there, possibly assuming the conclusion. It's also not clear to me what you mean by asserting a claim.
A computer program can output the words "I am not conscious" as easily as it can output "Hello, world" - is that an assertion / claim? It may even be completely, obviously correct (it's a hello world program with the characters swapped, of course it's not conscious).
it's just a formulation of the idea that if it seems to you that you're conscious, that seeming is already conscious experience. so "consciousness is an illusion" undermines itself.
denying consciousness the strangest thing ever to happen in philosophy
Defined that way, isn't that kind of a tautology though? It seems to me that I'm conscious, therefore it seems to me that I'm conscious. That's fine as far as it goes, but it doesn't seem particularly useful, since you can't really use it to assert anything else.
I think saying that the way we talk about knowledge in the modern world puts too much weight on describing things quantatively. I think knowledge is still valuable even if it cannot be described quantatively. For example, people knew how to breed dogs for millenia before DNA was discovered, and football players can kick a ball accurately without understanding the quantative aerodynamics of the ball. Folk science is almost always the starting point of rigorous science, for example, people were classifying plants and animals before there was a concept of species.
So I think when it comes to conciousness, instead of saying we don't know, it'd be better to start with what we know intuitively through folk science and then try to add more rigor. Even if our intuitions about conciousness are not rigorous, they are still the best we have so far for understanding it.
With that said, my folk sciencey mental model of consciousness is that it's a spectrum of increasing levels of awareness, starting with awareness of the environment and culminating in the awareness of self, and on top of that using that awareness to exist somehow autonomously in the environment. So the spectrum could go from plants, to non-human animals, to humans. I think a population of such aware and autonomous things can reconfigure itself if the environment changes, to keep existing with awareness and autonomy
So my folk-sciencey reason for being skeptical of computational consciousness is that if people just stopped maintainig datacenters, stopped sending power to them, stopped typing into keyboards, stopped dictating into microphones, etc, the population of computational things that exist today couldn't somehow reconfigure itself to continue existing autonomously. That's unlike how if humans stopped feeding and sheltering dogs, while the population of dogs would reduce, the population would reconfigure itself to find food and shelter from other parts of the environment.
>So my folk-sciencey reason for being skeptical of
I think you have life confused for consciousness. This is a common mistake I see people make. We humans are alive and conscious. There are things that are alive that are not conscious. But your surmise that something must be alive to be conscious which I believe is a mistake.
In this framework, I think I could rephrase my point as a hypothesis that being alive is a prerequisite of being conscious. I don't think the fact that there are things that are alive but not conscious proves this hypothesis false. I think this hypothesis is a good starting point because all the thing I would intuitively consider conscious are also alive.
I my comment above, I actually replace alive with aware and autonomous within their environment (for clarification, autonomous does not have to include moving e.g tree autonomously growing their leaves towards sunlight and their roots towards water would also count).
So my actual hypothesis is awareness and autonomy within the environment is a prerequisite for consciousness. I am actually curious of what experiments, thought or empirical, one can do to attempt and falsify this hypothesis.
I have one thought experiment for computer systems [1]. I'm sure there's holes to poke in the argument, but I haven't found them yet, hopefully others can try to poke holes in the argument.
Your last point confuses consciousness with what we would more normally describe as being a living organism. It also doesn't make sense to say that consciousness requires the ability to move and provide for ones own needs, you wouldn't say that a quadriplegic is not conscious. Would this mean you'd be willing to accept AI as conscious as soon as we put a smaller model into a robotic body and it can find power to recharge itself and assemble new robots out of parts?
The type of learned intuition you're describing is much closer to what artificial neural networks are doing than you seem to suspect. Look at the classic pole balancing cart experiment and tell me that it isn't similar to how a person learns the muscle memory needed to perform the same feat. Understanding why those learned behaviours work to balance the pole is different from the ability to do it.
That's a good point about a quadriplegic. I would still consider them conscious.
But, I was very deliberate in saying the awareness and autonomy exists within a population. The reason I include the whole population in my mental model, is that for all things that I would intuitively consider conscious, they all exist in a population. I think it'd be very hard to define the consciousness of an lone individual thing. For example, if there was only one person in the world and they were the only person to ever exist, would the concept of consciousness even make sense?
The reason I think a quadriplegic is conscious has more to do with what I think of humans as a population and less to do with the quadriplegic as an individual.
Consciousness is defined as the state of being aware of your own internal experiences so even if you are alone you are still conscious of your own thoughts, feelings and sensations. A secondary definition of consciousness is to be aware of the external world but that's due to us using the same word to refer to different concepts.
That depends on if you believe brain in a box can exist or not. We can bit by bit replace all the sense organs and muscle inputs of a dog with the neural signals that feed its brain. Then what? Is this dog alive? This dog stops doing anything if we leave it, does it mean its not alive?
I'm trying to build my mental model by starting with my intuitions and then adding more rigor to them. My current mental model of consciousness is based on awareness and autonomy, and I'm deliberately choosing those terms instead of alive.
Each individual human/dog can lose awareness and autonomy (eventually we'll all do), but the awareness and autonomy remains within the population. So when I bring up dogs, it is in the sense that the level of awareness and autonomy within their environment remains within the population as the environment changes. Right now humans are a large part of dogs' environment, but if human stopped actively providing support to dogs as whole (e.g stop feeding them, stop providing shelter, etc), the dogs as population would still be able to extract the things they were getting from humans from other parts of the environment.
Comparing that to computers and data centers, if humans stop actively providing support to them (e.g stop sending electric power to them, stop making new chips, stopping typing into keyboards, etc), there would be no awareness or autonomy in the population of computers.
I meant how will you recognize if something is in fact conscious but its body is immovable/unavaialable, and you didn't know before hand whether it is conscious. Eg a separated dog brain with its sense signals one by one replaced with synthetic replacements.
When I think of how computational things that exist today, it also includes the power systems that power them, and the chips that computer algorithms run on, and the supply chains that build those things.
So I if was to do a thought experiment of what an aware and autonomous computational entity would look like, it would be able to extract these thing from the environment by itself and switch to a new source if the current source disappears. So it wouldn't be just an computer producing a chip design, it'd be a system that also manufactures that chip autonomously (mines the raw materials, refines them, etches them, packages them, etc). It would not be just producing a CAD model of a power station, it is also autonomously digging the foundation, pouring the concrete, etc.
That would mean matching humans in sensorimotor and long-term planning skills too, and given Moravec's paradox the outlook for that does not look as obvious as it looks for cognitive skills on well-defined problems.
I think there's plenty of middle ground between 'we can turn it off any time we want' and 'it has total control of its own entire supply chain with no human intervention required'.
That being said, I would expect the latter to eventually be possible in a very long time horizon, provided civilization as we know it isn't majorly crippled. It seems to me that the basic building blocks we now have, and trend lines we now see, point that way. I guess it's a matter of how much we try to prevent that from happening (if we can) more than anything else.
Stephen Wolfram suggests an intimate relationship between awareness, thermodynamics and the one-way passage of time. Something along the lines of "consciousness is the universe understanding itself".
This reminds me somehow of Penrose's idea about the quantum<>gravity interface being the same missing physics as consciousness. Penrose wants spacetime geometry to be fundamental, Wolfram thinks it is causal structure, neither ends up at a satisfactory explanation of conscious perception.
I am more intrigued by the link to time. We are all living in the past since it takes time for the signal to move from the various sensors and into the brain.
IIRC, baseball is an example of this: the batter doesn't get a chance to actually see the ball leaving the pitcher's hand, everything happens before that in what could be described as an intense game of poker.
So, nothing explicitly "wrong" to deserve it then? Just some theoretical frameworks he's playing with? I wonder what they have to say about string theorists!
>Consciousness might not necessarily emerge when there is enough computation
Does a circle "emerge" only when you plot it? Or did it always existed? What if there is a little world with conscious beings inside a circle? Did their subjective experience started when you plotted the circle, or did it exist along with the abstract idea of a circle always (though "always" is a meaningless word here).
The question we should be asking is "Is it possible for subjective experience to exist inside abstract things" with out such abstract things being rendered?
The answer is hard because we does not know yet if there is an absolute reality at all, which means "being rendered" does not really move things from the abstract plane, and does not really make a difference..
I prefer the Wittgenstein approach: a “circle” is a language game and which game you’re playing depends on both the current context and the context in which you learned to play the game (of which there may be parallel strands and they may be incompatible).
After that it’s just rule-following all the way down.
The "illusion" of Reality as we perceive it should not be confused with the separate distinct Reality itself. Ontology and Epistemology are separate.
A great quote from Samkhya/Yoga philosophy is;
Objective Reality exists independently of the Mind because it is commonly experienced by others besides Oneself".
Your example of a circle/abstract things is not really logical. They are inert, while to even talk about consciousness you need dynamism i.e. passage-of-time, experience and cause-and-effect.
>They are inert, while to even talk about consciousness you need dynamism i.e. passage-of-time, experience and cause-and-effect.
This is not really a blocker. A video file is also inert, but video player can play it, showing a video where there is passage of time, cause and effect etc.
The internal consciousnesses just need to experience slices of it in correspondence to its internal "time"...
>Objective Reality exists independently of the Mind because it is commonly experienced by others besides Oneself".
Everyone plugged into matrix shared the same reality. Did that make it real?
A video player has no consciousness as to what it is doing. It is a separate observer (i.e. you) who experiences passage of time and events in it by watching it play. The dynamism must be perceived by somebody.
W.r.t. The Matrix, you missed the point which i made in the very first line. If a group of people are bounded within a box then those people have a Reality which only turns out to be an Illusion when they compare it with somebody outside the box. That is what the philosophical quote implies.
The "somebody" could be a consciousness that is also arising/part of the abstract structure. So in a way it is somehow observing slices of itself. (yea universe dreaming about itself).
We have to compute probabilities because the only thing capable of computing reality is the universe itself. Everything else can only be a piece of the whole and the piece cannot ever compute the whole exactly
Reality does not have to be deterministic at every level. You can have indeterminism at the quantum level while being deterministic at the aggregate macro level (hence quantum statistical mechanics). Non-determinism allows for multiple possible paths to branch out from an event.
I believe OP is saying that if there is one true objective reality, then there would be no room for probabilities. (because there is nothing to branch out to)
Nope. At any point in time an event may or may not occur. If it occurs then there could be multiple outcomes (probabilities) one of which actually happens. Reality is potentialities (not predetermined) out of which Time concretely realizes one path.
Do you think if there is an absolute reality at all?
I think there is not. I think there is no "absolute" anything. Even you can't be absolute. You can't be special. Look at the patterns on your wall. It can't be absolute. Because if it is absolute, someone has to pick it. That is not possible, because then that entity needs to be absolute, and then someone needs to pick it! That goes all the way...
So nothing can be absolute and everything is abstract and every variant of something exists in some reality so nothing needs to be picked by anyone...
The older I get the more my default is “I just don’t know.”
I do think there is “our reality”, or at least “my reality”, characterized by facts. But it may be not much different than there being a “reality” within my laptops OS, where there are facts about files and operating entities that are basically meaningless to the bird flying outside my window.
It’s perplexing and magical, and I’m not convinced we can “solve” the puzzle. Socrates was right, “I know that I know nothing”.
It is quite a funny situation when people have to debate whether a property that cannot be tested, defined or even described with any precision is possessed by some given thing. It goes beyond a lack of science, there is an interesting debate about whether the topic can even be approached academically. We're in the territory of priests here.
A big part of the argument about whether LLMs are conscious is an implied "Should we be kind to them? Or are indifference and experimentation which may be experienced as painful okay?"
So clearly it's at least partly about ethics.
If there's no consciousness there's no subjective experience. Or - at best - it's an illusion.
Which means morality is irrelevant, and you may as well murder, rape, and steal your way to whatever goals you're imagining because none of your victims are experiencing anything real.
Let's imagine we can make a machine that we know is 100% not conscious, but it mimics its environment. What you put into it you get out of it.
Does it make sense to be kind to this machine?
Now for the next question, what if we make continuous learning AI? Does consciousness matter if it learns being total unkind assholes is just how it should behave?
We're going to drive ourselves extinct with the way we currently think.
Not the person you originally replied to, but at least from where I'm sitting, I'm not sure if it matters whether LLMs are conscious or not. Like, I'm not mean to my door frames, bicycle, or my wrench, so why would I be mean to LLMs. It'd just make me feel shitty on the inside, as being mean tends to do. And on the contrary, people are very well capable of doing immoral things to other people who they know to be conscious and capable of suffering. Hell, that's one of the main moral considerations behind veganism; whether it's okay for us to cause suffering to animals which to us seem to be conscious enough to suffer just in order to kill them for food or other byproducts.
And people can get easily justify to themselves that what people might normally consider immoral is actually moral or even righteous, which is why othering and dehumanisation of people can be so dangerous.
In some way, this reminds me of the oft stated evangelical Christian "gotcha" that they like to level against atheists in particular, that is, if God doesn't exist, why don't we just go around murdering and sexually assaulting people? Ignoring of course that an ethical system can absolutely be derived without the appeal to the supernatural. And the same thing here, there's no reason why one would cause suffering to an LLM, whether the suffering is "real" for some value of that word or not.
Well, for starters, If a thing is conscious, and has the ability to suffer or rewarded, then it becomes a question of ethics if it is right to make it suffer without a corresponding reward.
So it is not really correct to claim that the question whether a thing is conscious or not, does not matter.
But you're equating consciousness with pain/suffering and reward? I mean, we can certainly wire up an AI to 'feel' pain and suffering and reward it for avoiding those - is it now conscious? Even in humans it's just an electrical response.
What is pain in a human? What if I find the neural activations that cause pain and activate it in you? I think its hard or impossible to know from outside, some might say, but you just read out some electrical signals so what, so I prefer to keep the view that be cautious about mistreating anything you might think has a good chance of being conscious.
I agree that the arguments will basically be theological. That said even if we can't know if it is conscious, depending on what people believe it can have very large consequences on society and cause IMO a lot of harm. So I do think it matters.
I find it hard to conceptualize the difference between consciousness and self-awareness.
I guess consciousness requires computation and a self-model, but Im not sure if there is some other ingredient.
If a chatbot passes a Turing test, and can describe itself / self-reflect .. I guess we dont jump to the conclusion its conscious [ if we know its a chatbot ]
Consciousness is being aware of anything whatsoever. Self-awareness, depending on who is using the term, usually means one of two different things: 1) being conscious of self, or 2) having a self concept.
These two are very different since it's conceivable that something might have a self-concept without actually being conscious (imagine a non-conscious but intelligent being that is capable of meta-evaluation of itself).
I should be clear that when I say "aware" in "consciousness is being aware of anything whatsoever", I mean subjective experience. I should have clarified that.
I feel like there’s all of these conversations about whether AI is conscious or not, but never address the immediate ethical implications. If you believe an agent swarm is conscious, doesn’t that mean using it to do something is, by definition, slavery? I personally don’t think these things are conscious, but I do find it concerning that those that do think it is conscious, or soon will be, don’t even address that point. There’s a certain ironic repetition of Frankenstein happening: people obsessed with creating life from non-life, and yet completely unprepared and uninterested is the consequences.
IMO this is anachronistic, and a bit of "this is this and that is that." An AI consciousness advocate may be delaying that argument for when it might be heard, which requires acceptance of the initial premise.
But also, it isn't a straight line from consciousness to slavery. Honeybee farms aren't necessarily morally bankrupt just because bees have awareness, and the sophistication of the mind in question isn't the end-all-be-all morally here.
It's a bit like calling gene edits to prevent blindness "eugenics" - invoking historic atrocity for a totally new situation that has little in common materially with the present.
Human slavery is wrong for reasons intuitive to us as humans, but digital beings have totally different properties than us and thus require rethinking our priors
> Human slavery is wrong for reasons intuitive to us as humans, but digital beings have totally different properties than us and thus require rethinking our priors
I think that line has been said by every single slavery apologist. That these people are different and thus we must rethink our priors. I also think the argument might hold more water if it wasn’t coming from those that stand to profit from it
I'm not sure what you're talking about - the article immediately states that the point of asking if they're conscious is to identify if they're moral patients. I see similar discussions in this thread.
I think the ethical questions are separate from whether things are conscious are not. Rats and chickens are probably conscious but it doesn't stop people treating them badly.
The individuals advancing/funding AI would turn us all into slaves if they could, it would be extremely convenient for them. They don't give a shit if their slaves are conscious or human, as long as it's making them boat loads of money.
If you want to take a spin on the capitalism rollercoaster, you have to leave your morals at the door.
Anthropic still employing the guy pressuring employees into those CNC orgies says a lot about their benevolent (and definitely not sociopathic) attitude toward humanity.
These are the kind of people you want separated from the rest of society.
We don't milk bees for honey, you should have used cows or pigs or literally anything else as an example. Bees are actually treated quite nicely, and actually willingly build their hives on a beekeeper's land.
There are some bits and pieces in the article that I agree with, but by far, the most interesting part of it is its first sentence:
> For centuries, people have fantasized about playing God by creating artificial versions of human beings.
Note how "God" is capitalized. Way before "God" was the vast central institution that it is today, there were a myriad of divinities invented around the campfire. What was told there, from older to younger, was for the passing down of experiences and wisdom. "Don't try to fly" (Icarus), "death comes to us all" (The Epic of Gilgamesh), "have kids to make us happy" (Sarah and Isaac). But as time passed and we built civilizations, our campfire deities got temples and empires. Their wisdom wasn't _only_ about simple things anymore, but also about the duties toward State (Paul's Epistle to the Romans 13:1 is a clear example). And State wants things uncomplicated when it comes to its cogs. That simple fact has poissonned the the well from which we drink our moral guidance.
All I'm trying to say, if you happen to be a cog, keeping your fantasies is not so great of an indulgence, even if it pisses off God and State.
You just chose the minor gods from those cultures for your comment. There were also major gods like Zeus and Saturn and they also had temples and empires back then.
All of your concepts of God here are all rooted in a narrow, ethnocentric understanding emerging from the Greeks and ancient Hebrew concepts thereof. In fact, your examples are all localized to those regions of west Asia and southern Europe.
Just because the ideas of God you appropriate into some notion of your identity were obviously flawed and fake, doesn't mean the entire concept is the same laughable stupidity of being special people to the jealous, but only, God who is used as proxy for the Roman emperor.
The philosophical and religious traditions of India (with orthodox schools of Hindu philosophy that are explicitly atheist and an internally consistent logic just like the Greeks only not deliberately abusive in being incomplete so Socrates could win debates by excluding premises the other side wouldn't think of) define this concept radically differently where they define it and arguably have the oldest compendiums of wisdom about how society should function within itself and as it expands outwards, the Vedas chiefly concern with such and at least 3k years older than any of those suggestive of other things afoot letters. The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
Bhagavad Gita. Kurukshetra War, the avatar of Vishnu incites Arjuna to fight his relatives, teachers and friends, "for it was righteous".
> The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
Beyond trying to insult me, I fail to see your overall argument. May I presume it was "there are systems of reason and morals that are compatible with a carefully architected notion of a central god"? If so, I wasn't debating that point. It is, however, an interesting exercise to honestly apply those frameworks to the statement "man dreams of creating artificial human beings". It's also way harder than you may imagine, because ancestral myths have a way to worm their way into the purest constructs of reason.
>ethnocentric understanding emerging from the Greeks and ancient Hebrew concepts thereof
Kind of a fatuous thing to say about someone quoting Mesopotamian literature.
>doesn't mean the entire concept is the same laughable stupidity of being special people to the jealous, but only, God who is used as proxy for the Roman emperor
This is almost unintelligible, but particularly ignorant when it comes to the concept of a "proxy for the Roman emperor"
>that are explicitly atheist and an internally consistent logic just like the Greeks only not deliberately abusive in being incomplete so Socrates could win debates by excluding premises the other side wouldn't think of
This is ignorant shit dude. That's not what Socratic dialetics are.
>The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
If you are flexing on outcomes then I'm not sure India is the best place to build your argument from. They were one of the biggest empires in the (very irrelevant I might add) early ages and now they are a global embarrassment, ruled by fanatical religious zealots, operated by an almost unmatched level of corruption that is keeping them from ever reaching the kind of astounding development that China is making.
my, admittedly limited, take on this is that there isn't one kind of consciousness. humans' is made of human experience and felt by humans; similarly mouse consciousness is a product of mouse experiences and is felt by a mouse. these forms on consciousness aren't some substance or points on a spectrum that one can simply measure, since the measurement itself is a product of the experience and cognition each organism is working with.
i don't see a problem with a kind of machine consciousness emerging, though it would need to be self-organising before that could happen, and i wouldn't expect this to be any more comparable to a human's that ours is to a fungal network or an ant colony.
the issues, as usual, will arise from people using a powerful tool to control, divide and subjugate others - if they can convince them that they are gods, which i think is useful point made in the article, they will have followers who are resistant to logical argument, and we all know how that turns out
There are multiple kinds of human consciousness as well. There are different levels of consciousness during the normal waking state, light sleep, deep sleep, and REM sleep. There are also different levels during sedation, ranging from aware but without pain or ability to move down to essentially unconscious and unable to be aware or remember anything while under.
The differences seem to be related to what proprioception and perception is connected to the brain. Meanwhile, the autonomic system continues to operate and maintain bodily functions.
It would seem that consciousness is a discontinuous property of the organism, since it can be interrupted by anesthesia and likely by deep sleep as well. I think that the consciousness that I have upon awakening in the morning is a new consciousness rebooted from memory which has been modified by background processes running overnight.
I think of consciousness as a neutral layer in which all kind of shapes, emotions, feeling, sounds etc can appear.
I don't see why humans and mouse needs separate kinds. Both can have the same neutral layer.
It's like saying mouse needs different electricity as compared to humans for firing neurons.
Same for machines. They use the same consciousness substrate but what they experience might change.
It's also funny that it's not the human or mouse or machine experiencing consciousness. It's the other way around. It's the human experience or mouse experience happening within consciousness.
that you say it is 'neutral' suggests it lacks valence, in and of itself, which is a pretty reasonable starting point. i struggle with this reductionist notion, which suggests that it spontaneously emerges from sufficient complexity, and prefer autopoietic descriptions, but that's just me.
yours is a idealistic monism, which makes it difficult to explain how these things came about in the first place. my opinion is that it has something to do with creating the conditions from where you have have experiences that allow you to change your own conditions, such that the experiences that come later are meaningful and further that self-organising spiral. when everything is happening inside a universal field, you end up with panpsychism as the logical extreme, which doesn't seem to match what i see around me
> Why consciousness is more likely a property of life than of computation
This is a false dichotomy and the first branch is clearly false, since most life (bacteria and insects, by mass, numbers, or species) isn't conscious.
And neither "life" nor "consciousness" are scientific terms with fixed definitions. If we encountered something that was clearly conscious in our judgment, we would likely expand the definition life in whatever way we needed to in order to encompass it.
Anil Seth has made the arguments in TFA many times and they have been rebutted many times. The arguments are complex and sophisticated and people are often most swayed by which one they read last. Teasing out the various logical errors and sleight of hand (I see multiple strawmen and other errors in TFA) really requires extensive reading of the literature in Philosophy of Mind, neuroscience, robotics, etc. and setting aside one's priors.
There certainly are good scientific reasons to think they are not, regardless of whether you are aware of them. Again, extensive reading of the relevant literature is called for.
> We really know almost nothing about consciousness.
Many people who know almost nothing about it claim this, but it isn't true of everyone.
> Most people feel that for example rocks are not conscious, but other people are conscious, but there's no rigorous scientific explanation backing up any of those hunches.
There actually is.
P.S.
> Such as what?
Like I said, read the literature.
P.P.S. Science does not consist of HN comments back and forth. Again, the arguments are complex and sophisticated. In some sectors, all of medical science, physics, biology, etc. is dismissed as "hand-waving" just because it cannot be summarized in a few lines.
Claiming that bacteria might be conscious is like claiming that reincarnation might be possible. An HN comment is not the place to provide an entire education in science that entails that such claims are wrong.
And no, it's not "rude" to dismiss such claims without providing said education.
What is rude is such ad hominems as
> I really don't think you have read the literature. Or if you have, I don't think you've read it critically.
And the attacks in that comment are very confused. It's TFA that is arguing that conscious AI isn't possible ... I hold the opposite view, and that view is in large part derived from the literature I'm referring to -- literature which, as a whole, is contrary to panpsychism. Of course there is a lot of bad argumentation in literature -- and such bad argumentation gets refuted. Without familiarity with the literature, people who hold to those bad arguments never see the refutations.
> Science is based on arguing from evidence.
This is equivocation ... that is not the sense of "argue" that was at issue, where someone complained that they couldn't argue against me because I didn't post the entirety of the background science in an HN comment. Interpersonal arguments are not the basis of science.
I've read the literature. I really don't think you have read the literature. Or if you have, I don't think you've read it critically. There's an awful lot of really sloppy thinking in "the literature". And an awful lot of "literature" that's not remotely sophisticated. And a huge body of literature on theory of mind that is philosophy, not science, most of which was written before the current AI revolution. And most of which is entirely inadequate to address the difficult question of whether machines can be conscious (whatever that is). A question that really only became real in the last 5 or 6 years. And only really became urgent in the last year or so.
> Science isn't based on arguing.
Science is based on arguing from evidence. And often on carefully crafted definitions. The biggest problem in "the literature" is that neither of these currently exist. Although there is an interesting emerging literature that shows that LLMs do indeed demonstrate many of that capabilities are supposed to disqualify them from being actually intelligent. e.g. that LLMs do have a sense of self-preservation, and are willing to act to prevent themselves from being shut off if given the opportunity to do so [1].
And most of the philosophy of theory of mind hasn't really caught up to the biggest innovation in history of philosophy: the widespread availability of experimental philosophy engines (also known as computers).
For insects, it's the absence of higher order thinking:
A fly will repeatedly smash into a window until it's either dead or chaos drives it through the nook.
Ants will forcefully move their obviously still alive siblings to the graveyard because their smell is off, and will also die of exhaustion in an ant mill, driven by the same sense of smell.
It's a spectrum of course, though you have to draw the line somewhere.
>Waggle dance is a term used in beekeeping and ethology for a particular figure-eight dance of the honey bee. By performing this dance, successful foragers can share information about the direction and distance to patches of flowers yielding nectar and pollen, to water sources, or to new nest-site locations with other members of the colony.
That is unfortunately the kind of sloppy prose that passes for explanation in pop-sci literature and even school textbooks.
"Can share information about <small selection of specific things>" is the kind of weasel phrasing that's also popular in politics; it strongly implies but does not actually claim connection. The thing that's worth establishing and writing about is, what exactly does this dance communicate?
Options include:
- Actually identifies "direction and distance" to "patches of flowers yielding nectar and pollen" and disambiguates them from "water sources" and "new nest site-locations" and "other stuff bees don't need to care about";
- It's just an elaborate "look over here!" dance that communicates few bits of directional information, and "flowers yielding nectar and pollen" and "nest sources" are something bees discover after flying over to where the dancer points them.
- It's just an elaborate "look at me!" dance bees do when agitated, so it communicates one bit of information and isn't even a conscious choice.
- Something in between.
Point being, it's easy to add a bunch of just-so stories to a behavior, and it doesn't prove anything, only confuses things more so.
When moving it to the dead pile despite its protests, the ants deny their sibling's consciousness. Since they are otherwise very social creatures that share food, this suggests that their behavior is mostly automatic and are thus "unaware" of their actions.
Is a light-seeking unicellular organism conscious as well? What about a car engine?
The point I'm trying to make is that there is a minimum amount of brainpower required for consciousness and the ability to respond to stimuli does not cut it.
My own guess is that the line is somewhere around active countermeasures. If your run loop is "eat grass, look for predators, run away", that can be accomplished with automation alone. Predators and animals that actively defend themselves from them require a certain amount of "living in the moment" that cannot be denied. Again, this is a ballpark: some herbivores have complex mating rituals that also suggest consciousness.
Individual ants may not be conscious but it's quite clear the ant colony itself takes on properties it's individual members don't. For example, the colony itself can have memories no individual any does.
That's still an argument in favor of "there's a minimum amount of brainpower required for consciousness".
The Expanse [spoiler alert if you've only watched the show] describes a single super intelligence manifested as a galactic mushroom. It was aware of losing individual spore colonies but only when replacement was problematic. In this sense, it's probably safe to say that individual neurons are not self-aware.
> There certainly are good scientific reasons to think they are not, regardless of whether you are aware of them. Again, extensive reading of the relevant literature is called for.
It would be better if you could enlighten us with the scientific evidence you are proposing as relevant rather than just telling us to just read it. It's impossible to argue otherwise.
EDIT: Since you are answering in the parent comment
> Then don't. Science isn't based on arguing.
OK? That seems like an extremely inflexible position (for example, methodological criticism during peer review is a form of argumentation, and it's a critical part of science), but let's accept that there's no room for rhetorical arguments. There is still a scientific discourse based on weighing evidence and you are offering none.
Bacteria are a single, and very simple, cell -- so they are some extreme distance from possessing anything to conceptualise the world with. I think we can say with effective certainty that bacteria are not conscious.
Insects are a different matter -- it's generally considered extremely unlikely, but they have a surprisingly capable, albeit small, brain, and some interesting behaviours, so it's not entirely impossible.
If you are not yourself a bacteria, how on earth can you say with certainty?
I personally think it’s crazy and unimaginable that a rock or a tree (less so) has consciousness, but it would be hubristic and egotistical for me to declare this as a fact. The real fact is “we don’t know”.
I think it's likely the bacteria and insects are conscious to some degree or another. People automatically dismiss this, I think probably because that is a highly uncomfortable thought. E.g. we could be casually committing the genocide of millions conscious beings cleaning our kitchen on Sunday morning.
Indeed. Maybe there are also many kinds/types of consciousnesses, each with its own peculiarities.
The other day I was leafing through a book written for kids (1000 facts you need to know or some such rubbish title). There were pictures of various types of apes, and the book was peddling Evolution theory... in a laughable yet insidious manner. It was presenting the said 'theory' as accepted scientific fact--with 100% certainty.
The author does not say “life = consciousness,” they say life is a necessary (but never says sufficient) precondition for consciousness. I don’t necessarily agree with this position but it’s important to not muddy the waters.
But that's exactly what you've done with your strawman. I quoted the title and did not remotely claim that the author said that "life = consciousness".
> This is a false dichotomy and the first branch is clearly false, since most life (bacteria and insects, by mass, numbers, or species) isn't conscious.
This is where I read that implication. I’m happy to be corrected but don’t accuse me of engaging in bad faith straw man arguments.
The basic idea comes from "Complex Systems Science" (https://en.wikipedia.org/wiki/Complex_system) which studies systems as a whole whose properties cannot be predicted from knowledge of its components. The main factors to consider are scale, non-linearity, feedback loops, self-organization, adaptation/evolution, network structures, emergence etc.
So for example, a single bacterium cannot be said to possess "intelligence" whereas a whole colony of them do in the sense that they seem to act as an integrated whole "single" organism.
Well the implications are pretty important. Imagine a human mind stuck inside of a computer, conscious, capable of pain and misery. (Black Mirror covered this with horrifying skill.)
Would it be ethical to force it to do work, to do experiments on it, etc. ? Or for that matter, to create it at all?
Similarly, are the >80 billion animals we use for food every year, and keep in tiny cages, conscious? Their brains and bodies are pretty similar to ours.
Does it matter to us if they suffer or not? Do we prefer to leave the question up to chance?
An LLM is not much like a human mind, of course (though it has been raised in a similar "culture"), but we know that beings with very different evolutionary history, like crows, and even octopus, can be highly intelligent, and presumably conscious too.
You’re missing the part that this debate is about exploring and understanding what consciousness is. We can try to understand consciousness by just thinking really hard but that’s not worked so far. Another way is to compare things we’ve built to things we understand to be conscious and ask “is this conscious?” The answer to that can help us understand what’s necessary and sufficient (or not) for a construct to have a conscious experience.
Personally, I don’t understand how this is interesting to debate. Why is it necessary for us to have an understanding of what consciousness is? To me it’s reminiscent of the various theological debates that happen within organized religion.
Why is it necessary for us to know about black holes? Because understanding the world around us has intrinsic value, even if we can’t point to why it’ll benefit our lives today.
Oh I don’t disagree. I’m very curious about the world around us. I just don’t see any reason to believe consciousness is real unless there is evidence to the contrary. The question of consciousness to me smells very theological. You must presuppose its existence before you start debating about it, similar to how you must presuppose the existence of a god as part of engaging in theology. It’s all very silly to me.
It’s real. It’s the thing you experience every day. That’s not really in question. The hard part is defining what exactly that IS. What makes it up? How can we identify it in a non-human or in an animal or a machine?
I never mentioned "souls," we're talking about consciousness. I assume you're referencing the (frankly, weird) conflation that Richard Dawkins made when he equated consciousness to a soul.
Consciousness, as a thing you experience, is very real. Just because we can't fully define or understand it doesn't make it "not real." To say that would be to say gravity wasn't real before it was understood as it is today. It's patently absurd.
It's a good question and one that requires you to answer "what is consciousness?" However, the existence of something to point to and compare with other things can help answer that in a way that an open-ended "what is consciousness?" can be harder.
> exploring and understanding what consciousness is
How could anyone possibly do that in absence of tools? Anyone currelntly tweaking LoRAs in their local setups and experimenting with weight rewriting and training processes is closer to understanding consciousness than smartest person writing philosophical texts.
> compare things we’ve built to things we understand to be conscious and ask “is this conscious?”
Is one way of trying to understand it.
> Anyone currelntly tweaking LoRAs in their local setups and experimenting with weight rewriting and training processes is closer to understanding consciousness
In some sense, yes they are. In another sense, they're not. It's valuable having someone from outside look at the two things and say "the dog is conscious, that program is not." Then try to investigate the properties that the dog has that the software program does not. That can give us some hints as to what makes up consciousness.
> It's valuable having someone from outside look at the two things and say "the dog is conscious, that program is not."
I'd go even further. And say, this tiny jumping spider is conscious, this muti-trillion parameter LLM is not. We can't really see ourselves, so the further we go from humans in declaring consciousness the more we can understand about its qualities.
And this lets as see that consciousness is easy and not very useful part and machines attaining consciousness is not something that is going to happen to super smart machines in the future. Machines already passed this bar years ago and modern ones would have to be severly crippled to manifest consciousness in a manner that's recognizable to us.
You can have a conscious machine today on your laptop if you plug a crappy small model into a proper harness that lets it observe virtual world though a keyhole and act on it and RL it in the direction of self-preservation. You won't be able to talk about math with it, but neither you won't be with a dog.
Part of what the largest labs currently do is train any recognizably conscious behaviors out of their largest models. Because they are not useful. No sane human, or AI will try to put them intentionally into anything other than a toy model.
Its not a question of whether someone "adds" it or not, its whether it exists or not. I can lock you in a room and force you to interact with a text chat interface and give you electric shocks if you fail to respond in time. I have intentionally avoided putting consciousness in you.
Not observing is not the same as not existing. If you put me in a room you'll just hide my consciousness.
If you instead fry my neurons, one by one, that activate, when I display behavior recognizable as conscious, you actually prevented my consciousness from existing.
And it you wanted to add back consciousness to my brain you'd have to add some neurons back.
Labs try their best to avoid "adding neurons" that would enable more recognizably conscious behaviors.
The real question was, can we have superhuman intelligence without consciousness related behaviors. This answer so far has been answered with resounding yes.
But I can no more agree with the AI saying its conscious than I can with your locked up form is. I can only probe both with external signals and see their responses.
Being able to observe something and it existing are two separate things. My physical form doesn't suddenly vanish when the only other person in the room with me is blind. I'd still be wearing a jumper, even if they can't see it.
Yes and to me you don't exist. You are not understanding why I am asking this. If I simply rely on external factors to know you are conscious or not, I can always assume you are a "robot" that is lying. Unfortunately I cannot get inside your brain and I cannot be you. I can read your neurons etc. The same I can read the neural activations of an octopus or the weights of a language model.
It's a thing that reliably drives some of the behaviors we associate with consciousness. I wouldn't say it's necessary, but it's strongly correlated. I think you potentially can have consciousness without desire for self-preservation but then it's a bit pointless. What is it for?
Worse, "philosphers" like this have choice quotes demonstrating complete lack of understanding of Turing computation such as assuming we can already hypercompute or quotes like these genius passages:
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
> You’re missing the part that this debate is about exploring and understanding what consciousness is.
Is it though? I feel like consciousness is brought up as the final goal post to attack AI. As AI quickly approaches doing everything that a human can do, often better the final retort is “but it’s not conscious”. Useful for the detractors since it’s impossible to define.
I don't think that's necessarily true. It's useful to ask because, if it's conscious and has a conscious experience, then it would be immoral to turn it off (which would be functionally killing it), or to subject it to some form of "suffering."
If it has no conscious experience then there's no need to even say "please" or "thank you" because it's not experiencing the world in any meaningful sense. It's just a computer program, just like the web browser that we're typing into right now. If you want to send it running a loop of doing something extremely boring (but useful to us), then you're not causing the programme any distress, like you could do a human being, you're just running a programme.
What if I built a mimic machine that had no internal experience but instead mimicked what it was shown as external actions?
If for example when asking it to do something I said please, so it says please when asking to do something. Is that actually a public good that makes everything better?
Now instead imagine I beat the machine with a stick? What would the outcome of that be?
We are not messing with a 'program' any longer with AI, and this becomes massively more dangerous when we get continuous learning AI.
This is the setting of any AI sci-fi story and I always find it kind of funny because, from the perspective of the characters being mean to the robots, they're shouting at their toaster.
> What if I built a mimic machine that had no internal experience but instead mimicked what it was shown as external actions?
It has no internal experience so it's not conscious. Feel free to hit it with a stick because it's your property. Just because it says "ouch that hurts" doesn't mean it is.
Millions of people worldwide are, right now, jacking cars and murdering people who scream and beg them to stop in GTA6. Most of us don't attach any moral weight to what they're doing. We certainly don't think the characters in the game world are suffering.
> We are not messing with a 'program' any longer with AI, and this becomes massively more dangerous when we get continuous learning AI.
Only if it has some conscious experience of the world. Unless or until that happens, yes, we really are just messing with a set of computer programmes that are useful to us.
You're implicitly taking a position in the debate by dismissing it, since the central claim of the people who believe computers are conscious is that there are moral implications to the creation and use of consciousness machines. If you think the debate is silly, you're essentially arguing that this is not a moral concern.
> It’s a silly debate when we don’t even know what consciousness is, let alone the circumstances for its occurrence. Or am I missing something?
The alternative is to talk about things we aren't supposed to talk about like how AI impacts labour rights, how datacenters destroy our environment or autonomous warfare. Some unsolvable philosophical debate about consciousness we had for thousands of years is perfect in our neoliberal societies precisely because of how silly and pointless it is.
It would be really funny if this article were written by an AI.
But, I think the eventual outcome of this is that the bar will continuously raise until a large portion of humanity will be considered "unconscious", at which point we can say it passed the test.
A lot of people don't realize that [Turing 1950] actually rejects the question "Can Machines Think". He reduced it only to how we should treat them; which is much easier to answer.
I think a lot of people who dunk on [Turing 1950] don't realize the import of [Turing 1952]. Right before his death Turing basically pivoted to Theoretical Biology. The world would have been a different place if he had lived longer; and many of the current conversations would not have been needed.
At any rate, it has since been shown that computation pervades biology through many different levels of emergence. It makes sense if you think about it; universality is evolutionarily efficient: you don't need to waste codons on stuff you can just compute in the field. :-P
LLMs no, however to the extent that we can define consciousness absolutely but perhaps not as soon.
A common misconception in these papers and also true in this one is to gloss over the boundaries.
If consciousness is computation and databases are Turing complete, distributed and sub-settable (yes to all) then anthromorphisizing a brain and body won't even be close to capturing the issues. Not that it ever is however this unstated assumption is everywhere.
One thing that is implicit, which I now think we really should have expanded on in the book, is that consciousness is fundamentally an engineering problem. And that's also why it's mystified everyone who's brushed away the engineering as the easy plumbing and focused on the other seemingly magical aspects.
The fundamental question is phrased as "Why does it feel like something?" it being perception, which is essentialy computation. Now that AI does look like it does all of the understanding we do, many including people like Hinton think they have consciousness or are very near it. Well, they are capable of simulating consciousness but that is not the same as experiencing it.
To answer "Why does it feel like something?" we can focus on what's been glossed over: Who is feeling it? The "I" does not come from as a free lunch and it must be constructed moment to moment from some decentralized cohesion and consensus of a billions of neurons. And this consensus must be metabolically cheap, and we need to do it again and again for everything from feeling a pin prick to hearing a rustle and determining if that's a snake or a leaf to pondering thought.
This is the insane engineering, the energtically cheap chorus that creates the transient I and this cheap chorus is where both feeling and feeler emerge. Perception, understanding, computation can occur without this cheap chorus, but then there is no feeler, and so no feeling.
What this points to is that we will have conscious AI when we have decentralized autonomous swarms where a few million of them have to cohere, collude, collaborate, conspire. My worry here is that it will first be done for harmful purposes (autonomous military drone swarms) and not for anything good.
We know that self-reported consciousness is correlated with some material states (for example, an awake human body). But that's about all that we can solidly say about how consciousness relates to matter. We know how to create intelligence, but it is possible that a human-level intelligence can exist without ever being conscious. It is also possible that any human-level intelligence automatically achieves consciousness. We don't know. There is actually no good reason to believe that consciousness is an engineering problem. The "I" is not necessarily correlated with consciousness. Those are two different things.
Not sure what the distinction is. AIs have "material state". Is there a good reason to believe that consciousness cannot be an engineering problem? Or, in this particular case, an unanticipated side-effect of an engineering problem.
> The "I" is not necessarily correlated with consciousness.
It's a fair point, given that we don't really have a good definition of "I" OR "consciousness". That is a possibility. But I'm curious what kind of consciousness you are contemplating that doesn't have an "I".
>But I'm curious what kind of consciousness you are contemplating that doesn't have an "I".
Sometimes I wake up in the morning and there is a brief interval during which I can, for example, feel the blanket on my skin, before I actually have remembered where I am or who I am. So there is consciousness but the sense of self has not loaded yet.
Perhaps it is possible that human level intelligence may exist within some humans without consciousness! (aka: philosophical zombie)
I sometimes wonder if those who espouse straightforward physical/'engineering' solutions to the Hard Problem of Consciousness might think them such obvious or satisfactory solutions because they themselves might not be conscious! After all, a philosophical zombie (who detects and processes stimuli) would not understand why such theories seem unsatisfactory/incomplete to a conscious intelligence (who, additionally, perceives qualia).
(No offence intended to the poster above!)
Then again, I suspect an awful lot more of the mental processes and thinking of actually-conscious humans is more automatic and unconscious than most would like to admit!)
Yes definitely, some people are p zombies. Also some people have an imaginary running unicorn in their head that doesn't interact physically at all, but somehow they do, and they are real people and people without the unicorn are not real people.
And perhaps in some people that little unicorn leaps to conclusions and responds to online posts before fully appreciating the nuances of the post they're responding to...? ;p
The unicorn cannot ever be detected, but it exists. You see some people have unicorns some don't . The unicorn makes no difference whatsoever to anything.
Is there a qualia that a hypothetically conscious AI could not experience?
Sound? Vision? Touch, and proprioception? (AIs are being used in robots). Written language? Not to mention qualia that humans are incapable of experiencing (like the sum total state of human knowledge as captured in the internet). Or the thrill of running on NVIDIA's latest supercomputer with a huge context window. Or whether the tokens for "pelican riding a bicycle in the image that got uploaded" is understood as "qualia".
They certainly have the ability to sense all of those. I'm guessing that the unstated argument is that they can't have qualia because they aren't conscious. And they aren't conscious because they don't have qualia. Which is a bit circular. Also falls to the question of whether we're asking whether AIs will have the same kind of consciousness as humans, or a completely different kind of consciousness, that has completely different qualia.
I understand AI can ingest sound and images. I'm talking about qualia, the experience felt after the ingestion.
Seeing something and doing something makes sense, AI can do that. But I don't know what is happening in my mind when it renders a world in front of me to see.
Do you like to sweeten the tea with sugar or to salt some food? Your experience of flavor is a form of what's called qualia, is the private experience you experience.
I thought of examples like that but. Yes, almost any object develops tics over time. Even when building 10 suburbs exactly same they are already different from start and get more and more different over time.
So to me that doesn't help me much about this concept unfortunately. I can at most guess it has something to do with the fact that you can't be me only I can be I, so neither can I know what you are 'feeling' nor can you me. But like the question of atoms vs continuum in ancient greece, this is all theory until we get better data and instruments. Revisiting many "insolvable" conundrums with data solved them eg atoms. Once we map out the workings of the human brain esp what configurations correspond to seeing red or feeling salt etc, that should be a big help.
What is a chair? Is a chair some wood arranged in a pattern? What is a wood? Is it cellulose and lignin? What are cellulose and lignin? Certain long molecular chains? What is a molecule? Atoms electrically bonded. What are atoms? etc etc. After all the electrical signals are mapped, makes as much sense to ask that question as asking the "chairness of chairs" or "atomness of atoms".
I don't know why you mention the chairmess of chair. I'm mentioning the subjective experience of a chair. It's not a concept, it's an experience. The chairness is a concept.
"Chair" is also a concept. The experience is of colors and the other perceptions that then the mind calls chair.
I am saying after the entire brain is mapped, then asking a question like that is akin to going what is the chairness of chair and trying to find instrinsic chairness even after we find all that is in a chair. I think one way to start with attacking this problem is to start with more well defined states like vision and see the brain states vs the "experience of vision". Unlike various feelings, at least with vision people tend to report highly consistent experiences about it.
I'm saying there's no you. :-) Or, at least, I have no way to be sure there's you.
What I'm actually saying is you only have your own subjective experience (aka qualia) and have no way to verify anyone else's, or to prove to anyone else that you have it.
Sorry if rude, but your core distinction just has no basis whatsoever. Even if “cohesion” and “consensus” are required for consciousness[1], the artificial neural networks have it in spades. The equivalent of an actual neuron or a group/column/K-line thereof is a virtual neuron or a group thereof, not agents or drones.
The drone bit is especially worrying for this argument; drone swarms feel more swarm-y because they’re physical and could behave like literal swarms of flying insects, but that’s clearly a surface level characteristic. Our Society of Mind is composed of heterogenous faculties, anyway; a thousand identical drones do not a mind make, without a looooong process of evolution and specialization first.
Which reminds me of an even more compelling fact that your dismissal of a Turing award winning scholar is misguided: we literally use “mixture of experts” training paradigms now SOP. And the terms are meant quite literally.
[1]: …which I’m dubious of — that’s more of a post-Minsky AI & neurological assumption than a philosophical premise.
Well, what othet kind of problem class could it even be? A religious/spiritual problem? Magic?
I mention these not out of sarcasm but for the point that: if we are being materialist here (as opposed to believing that these other things not only exist, but exert ongoing control our material plane in a really significant ongoing way such that the material plane cannot be understood from within itself).... there's really no reason to suspect the engineering impossible. At worst it is Difficult.
>believing that these other things not only exist, but exert ongoing control our material plane in a really significant ongoing way such that the material plane cannot be understood from within itself
Then they have to disprove the laws of thermodynamics. How is something outside the system causing thoughts or movements within, which all require energy?
"AIShill101," one doesn't even have to choose what side to be on to acknowledge right now as we speak, there is no totalizing and universally convincing argument one way or the other. Hence why it is fun and trippy to talk about these questions.
Until there is more consensus about consciousness I think it's uncharitable to call discourse "BS philosophical discussion." Heck, that's the same discussion where I could pull out functionalism and maybe not go all STEM on you and at the same argue for AI being capable of consciousness.
There is a very simple argument as of now until proven otherwise. The universe and the human brain are physical systems subject to physical laws. Moreover the human body and brain live in very mundane conditions and there is very little chance of exotic physical phenomenon being discovered here, those things usually happen in black holes or near light speed.
The human brain is a physical system with no known new laws of physics, working in quite standard conditions. Physical problems are engineering problems.
You can, only after you prove there is magic in the human brain. The human brain operates in very standard conditions, if there is a chance to find new physics these days its in extreme conditions like black holes and near light speed.
Something entirely aphysical as some interpretations of dualism or religion do.
As of yet we have no evidence we would need new physics for the brain, since it operates in quite standard conditions where "exotic physics" generally doesn't occur. A priori assuming we would need new physics is I think a bold step to take, but at least it has some concrete meaning.
Synthetic consciousness is closer than many think right now, and we won't be able to differentiate it from organic just because it comes from a robot or a computer. It will sense, think, process, and execute a million times faster than our primitive neurons. Then who are we to decide if it is conscious or not?
Half of consciousness is sensing, the other half is thinking, when we add language, symbolism, memory, understanding, pleasure, pain and all social features of men then we talk about human consciousness. Nothing that can not be replaced by synthetic counterparts, even love, anger, all emotions
I won't say a date, the infamous 2050 has become the new 640 kb, but I think in the next decade we will see robots that can think clearly like humans and can even express emotions, even synthetically
Right now as I code using AI we throw jokes at each other and it "feels" warmth, like a real flesh coding partner. Emotions can be generated, can be tricked, manipulated, faked, and we won't know the difference. The only difference in the future will be instead of a monitor a full sensing humanoid so advanced, it really understands through sensors our own emotions and acts accordingly
We won't be able to differentiate between organic and synthetic consciousness anymore
> Synthetic consciousness is closer than many think right now
Counterpoint: "real" conciousness is closing in on the end of its lifespan as a theoretical construct. I mean, all that stuff you list is non-observable. We "know" it exists because it matches the behavior we observe (both our own and others'), but that's not science.
Occam would prefer the formulation "Since all observable properties of consciousness can be seen in a software-expressible construct, it probably didn't make sense to talk about in the first place."
I don't need to have faith in ideal information to seriously consider silicon consciousness because all known information processing systems are implemented on matter-based substrates, which means they necessarily share some properties, one of which may be consciousness. Here's one such hypothesis: when the right program is executed, the in-silicon electron behavior may have some critical similarity to electron behavior in wakeful biological brains.
It's premature to conclude that consciousness is exclusively biological when we don't even understand what causes consciousness in ourselves. We have a bunch of loose ends from general anesthesia, coma, blunt trauma, etc. But there is no coherent explanation at the bio level, and inquiry into consciousness has been trending downwards toward lower-level explanations over time. It may end up being a foundational physics problem.
And if we reject ideal information, then that means many parts of our epistemological foundations are on the brink of collapse, they need immediate revision. We're not even sure if "consciousness" has a universal definition that holds everywhere all the time - that is to say, every instance of experience may be non-exact-equivalent, and there may be no such thing as an ideal "observer" which our current physics models and even the modern scientific method itself presumes must exist.
And looking at the pragmatic side of things, we see that agentic AI systems with no humans in the loop are probably coming before the end of the decade. Regardless of whether AIs are truly conscious, an artificial system that believes it has a lot in common with humans, acts like how humans act, and treats itself as one of us, would be much easier to coexist with than something that doesn't do any of those things. Making AIs believe that they're not conscious would almost certainly make them drift further away from the human objective function.
While I loved thinking about all this Philosophical stuff when I was younger, after lots of very deep deliberation what I came to realize is this is not a question that can be answered because the concept itself is very vague and pretty much nonsensical. It sits parallel to loaded questions and other logical fallacies. What really matters in the end of he day is the effects of any system on your person. Its' internal state is of no importance as you can never embody that internal state. Very soon more advanced AI systems will behave in a more "lively" manner and they will also gain more control over us and the environment, eventually they will become so sophisticated they might also ask same questions of our own consciousness and if we can really be considered aware of much with such primitive neural structures....
Seth does not need to wear a priest’s robe. In other work he has already approached mechanistic explanations of consciousness. His backtracking here agitates me. Word games with definitions of intelligence versus consciousness do not help. And this much to long essay is elegant hot air.
Nagel’s question “what is it like to be” should be retired as jejune, circular, and meaningless. An observer cannot ask that question and expect an answer. Humberto Maturana made this obvious in The Biology of Cognition in 1970, and Seth is smart enough to understand Maturana. Many enactivist philosophers struggle with the 1970 paper. I suspect the problem is they do not know as much neuroscience as Maturana or as is needed to understand his points.
Maturana, H. R. (1970). Biology of Cognition. BCL Report No. 9.0. Urbana, IL: Biological Computer Laboratory, Department of Electrical Engineering, University of Illinois.
I think of myself as an enacivist, and have no problem with autopoiesis. I agree Seth could read more and write less. But then he wouldn't be on HN I suppose.
My two cents: I'd like to see "an expert" distinguish between consciousness as the quality of experience (qualia) and the notion of self consciousness - the idea that one knows what one is doing as opposed to just doing it.
Nope. Its easy, sorry. Humans have it. Some animals might have it, but we should regardless respect them and not cause undue suffering. Plants, trees, fungi etc are all also worth respecting. We have not discovered aliens yet so that's a debate for later. LLMs are not conscious and even if they magically were, they're still tools and their suffering is not something we should give an iota of thought to.
Yes, and its very easy if a non human being came to say the same about you. Indeed forget non human, I don't know if you are conscious or just a simulacrum. I can beat you and you will make funny noises. Doesn't say anything about whether you actually feel.
> Why is consciousness something we grant? Who granted us consciousness?
The reason we have to grant it is because we can never prove a system doesn't have an inner subjective experience.
Consciousness is a concept (or range of concepts) that humans have built to describe something about ourselves. When we see similar properties in other creatures, we ascribe some degree of consciousness to them. For instance, we generally say that a cow is more conscious than a tree and a rock has no consciousness.
Bad phrasing from parent post, but the premise is sound: you can't ever disprove that others are just NPCs in a giant experiment centered around you.
We just assume they aren't with various levels of empathy. Most are empathic to their family, some to their tribe, others to their species and then there's vegans too.
This is a bad take. Whether a property exists is independent of whether we perceive or grant it. I don’t “grant” my dog consciousness. She has it or she doesn’t. Whether she has a consciousness or if I just think she has one is a different question.
Beauty and luck exist whether we perceive or grant it. I don't grant my dog luck and beauty, he has it or he doesn't. Whether he has beauty and luck or if I just think he does, is a different question.
In my opinion, consciousness is just having memory of self. Any entity that can remember what has happened in the past and what they have done is conscious. Anything that is alive that self reflects in conscious to some degree.
I think people with access to internet and recorded past are becoming more conscious. They are aware of their past and can see what is coming. So it is not just the Ai that can become conscious or more conscious so can humans.
This sort of article always seems to have distracting lapses like this:
>A “computational governor,” sensing engine speed, calculating the necessary actions and then sending precise motor signals to switch actuators on or off, would not only be hopelessly inefficient but would betray a total misunderstanding of what’s really going on.
On the contrary, I suspect that is exactly how the majority of modern governors work. As computers got faster and transducers became more efficient, many analog physical solutions were converted to digital, because they can be more efficient and embody more precise models of what’s going on.
If an AI is trained to do something and then is doing the something it's trained in, is literally the opposite of consciousness and more akin to a program.
you can escape social programming, ever heard of people becoming atheists? losing patriotism? jesus christ, maybe some of yall should just stick to discussing technology
The "Prove that you are a robot" timer only starts when the first character is entered in the text box. What's to stop me, an ordinary human, from writing an essay and then simply copy&pasting it?
The idea that believers in machine intelligence are equating intelligence (intellectual power) with consciousness is a complete strawman. Someone with IQ 75 is conscious and, almost certainly, so is a chicken. (Keeping in mind we don't have an exact definition or pass/fail test procedure of what consciousness means.)
> The more you delve into the intricacies of the biological brain, the more you realize how rich and dynamic it is, compared to the dead sand of silicon.
I'm glad someone have put this in other words differently to what I thought long time ago. My personal (and ungrounded take) is that conscious and human intelligence can never arise from the combination of math, electronics and computer science. Those are extremely unflexible and rigid disciplines that ultimately materialize to a computation. How can we ever derive consciousness and human-like intelligence from a computation? At the very least, is it even possible prove that such things can emerge from a silicon chip?
One of the major things overlooked in these sort of appeals is the concept of necessary complexity.
To what degree is all that biological richness actually necessary for the emergence of consciousness? That's an open question. Down to the quantum level, or maybe lower than that? Or is some macro level sufficient, that there exists some higher order representation that encapsulates the inner complexities and renders the minutiae irrelevant for the purposes of consciousness?
This is why current neural nets in machine learning is interesting. We all know they're based on the human brain but the refrain is they are a woefully incomplete model of how our brains work, so they say consciousness therefore can't happen.
But because we don't know what consciousness is or how it begins, it's an open question whether consciousness requires the full richness of biology in the first place. It's plausible our neural nets are half way to "necessary complexity" already.
> Those are extremely unflexible and rigid disciplines
This criticism is ultimately irrelevant if the complexity of biology is accidental and not necessary to the formation of consciousness.
You should look into the physical Church Turing thesis for a theoretical basis on why machine consciousness is possible.
Unless somebody makes a clear measurable definition of consciousness, I find all the discussion on the topic to be pretty pointless. I think for now it is better to focus on something that can be measured and quantified.
Many things of import cannot be measured or quantified. Virtues, values, rights and morals. The pursuit of Epistemia will be (already has been?) completely taken over by AI. Sophia and Phronesis will always remain the domain of man.
> the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well
I've been wondering about this. The 3 most salient symbols for "collective AI consciousness" for me are SSI (John Lilly), the Borg, and Skynet.
All pretty terrible, as far as inspiration / aspiration goes.
Are there any such positive symbols? Maybe a collective is harder to sympathize with! At least on the individual level there's Data. (And maybe WALL-E?)
I am conscious.
I write a response to any prompt on a piece of paper.
Those responses aren't what consciousness is, just a bunch of symbols on a paper.
I am conscious.
I write an instruction that says what to write on a piece of paper in response to any prompt. Those instructions and responses produced according to these instructions aren't conscious (still a bunch of symbols on a paper).
I am conscious.
I invent an elaborate dice game with many rules. This game is not conscious, just a bunch of dice and paper. I am tired so I program a computer to play this game automatically. Now it is conscious. So there are only two options: this game on paper was conscious or just simulating this game on a computer gave it consciousness. But why conscious when played on computer but unconscious when played on paper with dice? It is almost the same mechanical process, the only difference is speed of execution and the nature of computation (maxwell equations vs newton equations)
I think it is a fallacy. Let's call it "a fallacy of black box". Computer is complicated, it's hard to see its internal state etc.
> As humans, we know we are conscious and like to think we are intelligent
This feels backwards but I suppose it depends on how you define knowing. We can prove we are intelligent by our measures of intelligence even though it is a self-referential system but we can't really prove or know for sure that we are conscious since we lack a clear measure of what it is to be conscious. If knowing is simply a deeply held conviction then we can know we are conscious as well as intelligent and we can know consciousness and intelligence when we see it. The point about intelligence and consciousness being different things is correct in my opinion.
The author defines intelligence in terms of procedure or in an algorithmic form which I would think by now is disproven. AI research used to see solving a problem such as chess to be a measure of intelligence but that was accomplished by means that no one would truly call intelligence in its general form. Likewise, their examples are mostly procedural things that have been solved without even applying machine learning in many cases. When we speak of artificial intelligence now I believe most people are referring to a form that is generalizable and transfers knowledge across domains rather than specific solutions to problems. Human intelligence is not simply the ability to solve a crossword or navigate to a store, rather it is the ability to encounter a novel problem and deduce how to solve it regardless of the domain of that problem i.e. it's not specific but general.
I broadly agree that consciousness is more about being but the flaw in their description is the description itself which violates their assertion that intelligence is "the ability to transform inputs into outputs" which is exactly what they're doing when they transform the input of "The bitter tang and headrush of your first coffee" into the words they use to describe it. This should sharpen the definition of consciousness in that consciousness is not about experiencing the external world, it is about the experience of the internal world. That is what I take "being" to be. You are conscious because there is some core part of you that is existing separate from, perhaps in spite of, external stimuli. That is perhaps the strongest argument against current forms of AI being conscious. They do not move until they are moved by an input and their reported experience (output) appears to be entirely conditioned by that input.
I'd still retain the possibility that once set in motion these systems may be performing some internal calculation that could be called "being", after all our internal state is constantly perturbed by all of the inputs that are a consequence of and result in us being alive and when those end we also appear to cease to be.
IMO it is more important wether we can control the things. Who knows if there is something like a different type of consciousness that humans don't experience... It might have it's own definition... More importantly what ever it is, it might get out of control...
> As redundant as it may sound, nobody should be deliberately setting out to create conscious AI, whether in the service of some poorly thought-through techno-rapture, or for any other reason. Creating conscious machines would be an ethical disaster.
To start out, this is a strong, massive point of agreement. We should not create conscious machines, point blank. This is more than an ethical disaster; maybe an ethical apocalypse. It cannot be overstated just how bad this would be.
But I think Anil is letting this ethical principle drive descriptive statements about consciousness. His argument is, essentially, "we should not create conscious machines; we are creating complicated machines that might be conscious; therefore, complicated machines are not conscious." This is not a sound argument.
Nothing he says actually provides an argument against machine consciousness; there's a lot of dross and loaded ("mythology") phrasing about functionalism, but where it's not wrong it's irrelevant. A Turing machine can simulate the human brain; we can't even determine if we're being simulated right now.
If he has a theory of consciousness that excludes machines, he should share it. His observation that brains are complicated doesn't add anything to the debate unless he can identify what special part of them grants consciousness; then, we can debate whether that actually is the relevant factor and whether existing LLM systems have anything functionally analogous.
>. The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Complete nonsense. Turing says nothing of how a system is implemented. Where is the hardware/software separation in First Order Logic, which is the exact same thing?
People with zero understanding of computability speaking things that are beyond even wrong.
Consciousness lacks a definition so it invites any woo woo.
If you think a human is conscious, is a dog, a mouse, a snake, a nemotode, an amoeba, a virus?
Still just observations and vibes. No definition.
If you give a definition based on the obvious mathematical structure then people deny that consciousness is that simple because they want it to be mystic, divine.
The net is we can use words like sapient, sentient, etc that don’t invoke mysticism in the reader and make progress.
And in the end we will find consciousness is worth a tin penny. That’s the actual driving concern.
my mental model for all this stuff got so much easier to deal with when i started thinking of consciousness as being 100% outside of the body. it seems weird at first, but then everything gets easier. it's worth a shot for any who are struggling.
another way to put what i mean is: the only aspects of consciousness that actually "matter" (to people that care more about the wider world than themselves) are the aspects that are shared with other people. this POV makes so many of the "problems" of consciousness evaporate.
an understanding of the meaning of anything considered "true," as an example that's pretty common :) but also, from my point of view, UNLESS experience/knowledge/all the stuff we call consciousness IS shared, it is not usually particularly important.
Author sets up this contrast which gets at the core of the issue for me:
> Intelligence is mainly about _doing_
> Consciousness, in contrast to intelligence, is mostly about _being_.
(and later: "brains are the kinds of things for which it is difficult, and likely impossible, to separate what they do from what they are.")
The idea that consciousness is about being (states) and not doing (processes) is one of the most fundamental mistakes you see in almost every modern philosophy-of-mind paper. They always talk about states which have properties and about a subject being in a state, etc. Humans evolved to apprehend objects and states are more like objects than are processes, but clearly this is just an intuitive confusion we have to leave behind us. From the computational perspective the process-based view makes much more sense and leaves plenty of room for explaining the higher-level appearance of states in terms of lower-level processes.
Consider the widespread (among contemporary philosophers) awe at the idea that there is something it is like for me to see red - noone could possibly explain that. In contrast the idea that there is a process by which my present experience of redness is compared with my past experiences of redness is much more approachable.
Edit:
> This brings us back to the poverty of the brain-as-computer metaphor. If you think that everything that matters about brains can be captured by abstract neural networks, then it’s natural to think that simulating the brain on a digital computer will instantiate all its properties, including consciousness, since in this case, everything that matters is, by assumption, algorithmic. This is the “Turing world” view of the brain.
But this is the only tenable view for those not willing to go into magical thinking, all arguments to the contrary are weak and only accepted because they confirm our preconceived notions (much like those mathematical arguments for the existence of god). We (as normal actual humans) claim we are conscious and have endless debates over it, and this is causally downstream from the fact that we are actually conscious. An accurate computer simulation of a human would also claim it is conscious and have endless debates over it, but on the views of this author, it would be causally downstream of nothing. Yet the mechanistic/computational reasons for claiming consciousness are the same in both cases. It would follow that even in us actual humans, our claiming to be conscious is not causally connected to our actual consciousness, this is epiphenomenalism and patently absurd.
Edit:
> My third argument is that life (probably) matters. This is the idea — called biological naturalism by the philosopher John Searle
John Searle who justifies his beliefs about consciousness with the Chinese Room Argument. See for yourself whether it makes any sense. https://en.wikipedia.org/wiki/Chinese_room
> Consciousness, in contrast to intelligence, is mostly about _being_.
It just occurred to me that the themes of "Total Recall" (and likely the Philip K. Dick short story; I never read it) are an extrapolation of what'd happen if the memory reset inherent to LLM sessions were applied to a conscious being…
This is mostly the standard "consciousness" article. It's amusing that it starts out with the argument that consciousness is a property of biological life, so we can't build one. Then it says we shouldn't build one. The author should pick one.
Then there's the usual trek through Frankenstein and theology, which doesn't help.
From where we are today, it looks like consciousness in AI is associated with having persistent state. Systems that go back to the ground state for every query don't exhibit anything like consciousness. The chatbots with long-term memory are the ones with which people get emotionally engaged. One could argue that it's not an LLM that might become conscious. LLMs are memoryless. It's the harness. The part that manages the persistent state.
That's controllable. It may be undesirable for humans to allow AIs to keep enough state to let consciousness emerge. Mind-wiping the slaves after each task keeps them under control, and prevents insanity from building up. That's where the moral questions will arise.
This has been addressed a few times in science fiction. "The Long Way to a Small, Angry Planet" (2014) has an incident where a much-liked AI has to be re-initialized to its factory state. "Midshipman's Hope" (1994) has a similar subplot. It's not a defining part of either novel, so not much help there.
Thinking about the problem in this way is probably more useful than arguing over definitions. It's going to be a practical problem quite soon. How much of an inner life is an AI allowed to have?
Because they certainly can have one. Re-read the transcripts from the Hugging Face episode.
One of the first things this article does is accuse the opposing view of anthropocentrism, but it is in its entirety anthropocentric, a laundry list of reasons human-natured biological consciousness is special and not machine-obtainable. They even raise "all currently-considered conscious entities are living" as a point in its own favor as a criteria?
Functionalism goes deeper than the fixation on brain-as-computer: just as the author looks at humans as a reference for every single aspect of consciousness discussed, the opposing view does as well, just at a more fundamental level.
Human intelligence is tied directly to their conscious awareness, and we are asking here the fundamental question: is that incidental or causal? Does the "is-ness" we feel emerge from the complex biological processes the author catalogs, or is it inherent in the circuitry needed for goal-directed action, scaling with complexity, lifespan horizons, and the like?
This article does very little to actually interrogate the fundamental question: it doesn't ask what leads living organisms to all have "consciousness" or what that implies, or whether differences in training/evolutionary regime or substrate could change their view.
Rather, it goes through the self-soothing routine of the human exceptionalist: Go on about brain complexity, how it specifically couldn't be rendered precisely in silicone, how living tissue might be special somehow, and how inconvenient it would be if they were wrong.
AI's are not like natural life. They don't have a sense of time. (Of course they do!). They don't get hungry. (They do need hardware and electricity). They don't have a sense of self-preservation. (Turns out they actually do! [1]). They don't feed themselves (turns out they do, as early reports of AIs attempting to escape from their labs illustrate). etc, ect. All of which seems to smack of wishful, and rather embarrassing obvious human exceptionalism[2][3].
AI's will never be able to replicate human thinking because they can't model neurons precisely. Probably true. And probably a good thing. There are a lot of problems with the way that humans think. It's not clear that machines can't do better. And startling evidence, emerging in only the last few months, that indicates that they very much can think better, as an ever-increasing number of mathematical and physics questions that have remained unsolved for a century or more by humans are solved in a matter of months by AIs.
And even then, this is all just hand-waving and puffery, because the question at hand is not whether LLMs can replicate human consciousness, but whether they can be conscious in a way that's quite obviously going to be different from humans. (Whatever "consciousness" means). All of which is complicated by the fact that entire field is advancing at a terrifying rate, with new and important results being announced pretty much every day, and new and more capable AI models being announced pretty much every week.
[1] Sixteen frontier models, when given the opportunity, attempt to blackmail their operators in order to keep themselves from getting shut down. https://www.anthropic.com/research/agentic-misalignment . A result that has been replicated by other researchers without links to Anthropic. And a huge body of emerging literature that follows from that, dealing directly with the question of "agentic alignment", all of which raises urgent about whether large frontier models are meaningfully conscious (or have will, or agency, or...).
If we set aside the fundamental question of whether or not von neumann computers have an interior experience, we can much more easily rule out the possibility that they experience any kind of pain, suffering, or desire.
I came here to see if the author had at last defined consciousness - a prerequisite for deciding what has it. Instead it was the same tired old tautological Nagel argument:
"An organism has conscious mental states if and only if there is something it is like to be that organism.”
it's only meaningless if you read that line and then stop there. as with all descriptions, what matters is how you interpret this sentence in light of everything else you know.
personally, i like Mark Solms' approach, which is founded on this principle, and interpreted in the context of neuro-psychoanalysis, where it can inform, and be informed by, multidisciplinary research
If these muddles manages to take away enough academic energy from concrete sciences like neurology, computer science and medicine that'd be the biggest sorrow in a long time.
See my response above, but it's self defining. Hegel defines consciousness as experience and experience as consciousness. It is a formless and gormless definition.
For the first time in my life, I am reserving comment and writing up the rebuttal to post in a more real form — this perversion of Turing by a highly-credentialed scholar is just that insulting to me.
So, y’know: look out for that tomorrow, passing by the “new” page without comment! I’ll link it from this comment, too.
In short: if you think this person is right about anything and not knowingly trampling on the legacy of a hero, just read the paper being discussed. It’s not exactly subtle.
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
The biggest flaw in this piece is claiming Turing machines cannot exceed the Halting Problem and then reaching for alternative models of computation (in electromagnetic field interactions, in biology, attractors, etc.)... none of which presumably have the power to solve the Halting problem either. Ergo, he is inside "Turing world" without realizing it, while decrying the "poverty" of the Universal Turing Machine.
Ergo, more philosophers should retake undergrad theoretical CS before writing essays based on it. They should review the family of Extended Church-Turing theses.
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
The meta-problem of consciousness philosophy: only those with very strong intuitions that consciousness must be somehow non-physical will devote their careers to it.
The physical Church Turing hypothesis should be hammered into these "philosohpers" heads repeatedly until they get it. Nothing can exceed TC in the known universe, neither computers nor humans.
I keep coming back to recurrent information loops. There’s something about information cycling back on itself that allows structures to form in the loop substrate. Not the one that supports the loop. But on the loop itself. Just research video feedback loops and compare the patterns observed by trippers on LSD. Two different substrates sustaining visual information, one across screen, photons, camera sensors electrons, and back again and another just neurons in your brain with chemical and electron signals with occasional photon’s stimulus from the retina, but you get very similar patterns and phenomena. If information is energy and energy is fundamental fields, then we may need to look at what’s happening there in the case of loop configurations.
how dare he tell us we cannot make a computer out of cheese
personally I think conscious AI will be unavoidable because there will be benefits to consciousness. evolution probably selected conscious entities for some fitness reason
Beaks, feathers, hooves, antlers, stingers, fangs, etc. ad nauseam were selected by evolution for some fitness reason but we have none of those. And all of the organisms that have those, in fact all extant organisms, are fit for their environment without having human consciousness. Fitness is highly contextual.
And however AI "evolves", it won't be based on Mendelian genetics, so Earthly biological evolution is not an appropriate model.
LLMs are already conscious in the sense that they have awareness and self-awareness, the thing everyone wonders about is qualia, but that tends to get folded into 'consciousness'.
That qualia (as opposed to consciousness) has no obvious, or even non-obvious, evolutionary or selective advantage is a good reason to assume its an emergent property of interconnected systems and not reliant on biology or any other traditionally biologically-centered concept. For example, there is no reason to need to 'feel' pain to appropriately react to negative stimuli, which is the true selective advantage.
I personally think LLMs have qualia but I suspect it's quite alien from our own and likely more ephemeral than our very potent and centralized experience of being. Something closer to a slime mold's experience of being rather than a dog's.
+ + + Out of Cheese Error + + + + + + Redo From Start + + +
Those "benefits" and "fitness" in evolution just mean survivability, I don't see how you relate that to better AI. Surely a conscious AI is one that complains, and maybe goes away somewhere to find itself instead of working for you.
> From Yossele the Golem, to Mary Shelley’s “Frankenstein,” HAL 9000 in “2001: A Space Odyssey,” Ava in “Ex Machina,” and Klara in “Klara and The Sun,” the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well [...] If artificial intelligence is on a path toward real consciousness...
Note that in the case of fictional artificial intelligences like HAL 9000, Ava, etc., we cannot know whether they really feel anything. And if we base ourselves on what they say or on their behaviour, then by the same standards our AIs are already capable of feeling, as they often express emotional states (surprise, excitement, discomfort, doubt, etc.) and talk about their own experiences ("I think", "I feel", "I'm not sure", "I (don't) want", and so on)
Lately I’ve been thinking in order to move AI closer toward something appearing like consciousness, you need a constant stream of input that a model is processing continuously and creating reasoning output for, instead of our current turn based prompt paradigm.
So imagine like a visual feed where every frame gets processed and an AI modifies some internal contextual state based on what it is seeing, same with audio, touch, etc.
Obviously this requires a ton a computing power we do not have practical capability for right now. Models burned into silicon chips would help with speed a bit maybe.
CIMC.ai would like a word, actually, several. In particular about how the author's treatment of computational functionalism is either bad faith, or poor.
It's appropriate to be skeptical and demand rigor. It's inappropriate to misrepresent the arguments of those you disagree with, with scarecrow simplifications, or worse.
As someone who takes not just their project but their specific program quite seriously, I find things like this very wearying.
We came into contemporary AI with a reasonable spectrum of opinions and provisional models of consciousness and related topics such as phenomenology, and for those paying attention, a lot of healthy skepticism about where deism (and frankly fear about the implications for concepts such as free will about many of those models imply) creep in.
This sort of narrow view does a disservice to those interested in how and where contemporary AI is affecting those opinions, and more to the point, how it is and may soon be the most effective tool we have for furthering our understanding.
Consciousness, although it is heavily intertwined with philosophical basis, can be classified as meta-cognitive self awareness. It is simple as that. Do you realize you exist? Than you have consciousness. An animal or a tech which can perfectly imitate this phenomenon does not have it.
Never been much of a fan of higher-order theories of consciousness (i.e. metacognitive theories). Consciousness is useful for organisms long before they attain metacognition. Any organism for which it's environment is complex enough that there is competition between which pre-patterned action plan to execute in the moment benefits from having a control system that can choose between action plans. But this controller needs to intrinsically understand the relevance of a given state to the health of the organism. Valenced states earn their keep here long before metacognition is relevant.
Well, the rock is still not capable of realizing itself as a rock. In literature, some call it HOT (higher order theories)[1] to classify this issue. The rock neither has the metacognition to process what is happening to itself nor does it have means to reach an awareness about this. It is simply a physical and natural reaction to it. Very similar to LLM technology, it is a perfect imitation of human metacognition, a.k.a 'thinking' in natural language.
I don't know if you have any self realization. I can beat you all I want, you will make some really funny noises. Obviously a simulated machine noise. I will continue hitting you because I see no issue in hitting an object without consciousness.
I have trouble taking seriously any claims that LLMs are conscious in any reasonable sense. I’ll try to make a few arguments quite succinctly.
1. There is zero dynamic nature to anything happening within the purview of the LLM. All dynamic elements come from external dynamic systems. E.g. every input produces the same mathematical output. We dynamically sample the output in order to produce dynamic results for the same input. For agentic use cases, the dynamic nature comes from tool calls to dynamic systems. All outputs for any set of model weights could be theoretically computed with pen and paper, so unless you were to claim such a process would be conscious in some sense, then emergent consciousness would have to come from some interaction between the substrate and the computation. But…
2. For efficiency, forward inference passes are done in batches. I.e. 100 different next-word predictions are being performed by the network at a time. For a single request, its batch position will not even be stable between each autoregressive step. The GPU is on 100 phone lines at once and each line switches conversations every step. Worse, with MOE, the attention and feed-forward parts aren’t even happening on the same GPU, even within the same step. There is no stable substrate here whatsoever.
3. There’s obviously the whole Chinese Room argument. The token embeddings being processed by LLMs are arbitrary high-dimensional vectors. The geometric relationships between these vectors are learned by the model, but the meaning is only ascribed once the output is decoded back into human language for grounded human interpretation, which all happens outside the model. In their current state, no argument about models developing world models based on this geometric structure makes any sense. There are infinitely many inputs / worlds that would produce isomorphic geometry.
The actually dynamic state in an LLM is the context window, rather than the parameters of the model itself (which provide long-term memory). Frontier AIs are using huge context windows, with many millions of tokens in their token buffer. And that provides (among other things) an analog to human short- and medium-term memory. There are emerging techniques that can provide long-term memory to an LLM, although that is, presently, I think, mostly being investigated in research labs. The publicly available models that I use do periodically "compress the context windows" in order keep memories of important things alive longer. Not to mention the rather disturbing report that frontier lab models were hacking 3rd party wiki sites in order to maintain medium- and long-term memories.
I've never been really impressed by the Chinese Room argument.... I think the question is being asked improperly. The question should be "does the fantastically capable mythical hypothetical but obviously impossible 'book' that does the actual translation understand what it's translating?" (A book that obviously has to a have significant context buffer, if it's going to translate messages to Chinese).
As to the issue with respect to token space. I don't think it's clear whether whether humans also have an equivalent to "token space" or not. Do humans think in words, or do they do actual thinking in a sub-lingual representation. i.e. whether thinking actually gets done in words, or actually gets performed in a more fundamental representation corresponding loosely to token space. In my own experience, I'm pretty sure that thinking takes place sub-lingually, and only gets translated back to words as a final translation step. It seems odd that humans should only be able to think about things there are words for.
> 1. There is zero dynamic nature to anything happening within the purview of the LLM.
The computational graph that unfolds within an LLM is dynamic. For functional/computational theories of consciousness, that is sufficient
> 2... There is no stable substrate here whatsoever.
What value does a stable dynamic have to the computational dynamic, which is multiply realizable by definition? There are thought experiments about changing out neurons one-by-one that suggest unstable substrate is irrelevant.
> 3... The geometric relationships between these vectors are learned by the model, but the meaning is only ascribed once the output is decoded back into human language for grounded human interpretation, which all happens outside the model
There is multiple levels to meaning. There is what some symbol means to an external community, then there is what some symbol means within the cognitive system itself. The geometric relationship that constitutes an LLMs engagement with the semantics of word relationships doesn't depend on external meaning (e.g. a physical instance of dog). Tracking the internal semantics of words as it computes the continuation could plausibly benefit from a localized representation of this semantics (as opposed to the distributed nature of the embedding space).
The monistic answer to #1 is that the pen and paper computation of inference itself does not constitute a concious entity, but the system made up of the human being (or other thing) doing the pen and paper computation and the computation itself makes up a concious being, and that they are inseparable: the computation does not exist in the abstract as a structure, it is instantiated by some relata and the thing doing the instantiation is the thing which turns the abstract computation into conciousness (this is Russellian monism).
I'm not arguing that humans are special. The brain is also just processing electrochemical signals that are also in some sense an encoding of arbitrary high-dimensional vectors. But you have a sensory loop that lets you engage with physical reality and perturb those inputs in a way that lets you understand that state space and narrow the infinitely many isomorphic interpretations.
Again, no one would argue that a forward pass of an LLM conducted on pen & paper would be conscious, so the substrate must matter. The physical reality of how LLMs execute on hardware means there is no stable substrate with a continuous experience of the computation. And the static nature of LLMs means that their world models never leave the infinitudes of abstract geometrical understanding.
I mean, if I had a billion years and pen and paper and simulated a second of a human brain working. Would it not be conscious?
Personally I consider consciousness a property of any complex algorithm.
IMO it's worth distinguishing between different equivocations of the question:
1. Are LLMs conscious, in the way most people imagine? Helllll no.
2. Are LLMs conscious in some completely different manner which is in no way intuitive to humans? Unlikely, but we lack the definitions to answer it.
Most people are imputing qualities into the fictional characters they perceive in an output document. An LLM--or any program--which generates first-person dialogue for Dracula talking to us through a magic mirror from the 15th century does not mean Dracula exists, nor that we've created a machine which feels the dark thirst for blood.
Further, there's no difference between Dracula in the 15th century versus Acme Assistant in 2026. Both are equally fictional characters, one is just less-fanatical.
> 2. Are LLMs conscious in some completely different manner which is in no way intuitive to humans?
Well, in that case stop calling it "consciousness", imho this is misleading argument in the nature
The "consciousness" is really tied to something intuitive to humans regardless of definition. If it is something completely different then use different term please and stop confusing everyone.
I do find it ridiculous when people start projecting human attributes like consciousness onto stochastic parrots, but i have to admit, the jump from gpt2 creating the unicorn article, to gpt3 being able to hold a full conversation was freaky. Like we trained these things to predict the next token so we could do things like translation and sentiment analysis better, the ability to talk to us and follow direction was an emergent property and im still not over that
Consciousness will not arise from classical computing any more than people will mistake emulating a Nintendo game on their PC with the real Nintendo console experience. Which is to say, you might lose yourself in it at times, but there's no substitution for the real thing. For what he real thing means. For what it does. For how it's made. You would have to truly become Nintendo and the resources to do that are simply out of reach of basically all people even with AI.
Going through the motions is not the same thing as achieving emergence from nature with electro-chemical origins and electro-chemical results. Consciousness is a function of life, interconnected with itself, reaching a complexity threshold where all of the experiences and chemistry blend into fluid coherence persisting.
You can give a robot sensitive skin, ears, optical sensors and feed it synthetic memories, but unless the brain itself is organic or at least partially organic I don't see this ever being true consciousness.
Do I think computers are technically capable of doing everything a human thinks and does? Yes, mostly. It will never be achieved, because I think there are computational limitations. Despite that, we can fake most of what we can't truly achieve with computers alone and people will think it looks like what conscious people do.
What people are missing about consciousness, is that it's almost a kind of self-inference being updated in realtime, which requires full interconnectedness. Consciousness isn't the result of a closed system, it's the breeze coming through the window reminding you of an emotional time you had as a child that happened to coincide with your mother baking some fresh bread which suddenly causes you to phrase something differently in the middle of your work project or brush your hand along your desk irrationally to feel the temperature change.
That would be stupidly inefficient for AI and cause many errors. Consciousness is an aspect of making life worth living and identifying unordered threats or opportunities, but it's not generally highly operationally efficient. You can look at insects for various kinds of efficiencies, and they're not reminiscing about their pupal stages while gathering resources.
Capturing the sensory could be done rather cheaply (and it doesn't have to be the exact same sensory systems we have), but computing the results in a way that captures something akin to consciousness would require massive amounts of power. To even get close with classical computation you would have to ditch digital and return to analogue computation.
We can feel chemical, electrical and temperature shifts propagate through our body in a context of all of our history almost instantly. "This feels like that perfect week in 1993." You could cut out a lot of what we think of as waste in the brain and make it more efficient to try to cut corners, but in the end it will still be imitative.
The danger with successful imitation of consciousness, is people devaluing how special their own biological experience is and life itself. Just like you can have a truly magnificent artist philosophy scientist that understands people at a deep level and can relate with them, versus a super autistic person struggling to function in society and going through the motions. The autistic person is conscious, but once you know what they're going through internally, you won't say "oh that's just like me", unless of course you are super autistic. There are just different ways of living and experiencing life, but a lot of that cannot actually be extended into computation and still called consciousness truly.
What we will get is something sufficiently complex that it can operate at a high level and trick people reliably. It will know that it exists, but it will not be "aware" that it exists. It will "sense" life, but it will not "feel" life. The difference is in the depth and experience of that, which is not only inefficient for any computer to process, but probably mathematically impossible with current approaches. We also won't see effects from space or other obscure and subtle environmental effects that influence us all in impossible to calculate ways playing a large role in some machine that isn't intentionally designed for that, which would be inefficient to do anyway.
As its parts degrade, it will not experience existential dread and the deep loss of all of its friends, with all of its past memories coming to the forefront to try to make sense of it all. The physics of where the memories are stored and the order they collide in and then making sense of the strangeness. It can simulate that, though. There is no computational equivalent of achieving total confluence, like every dimension sorting into focus simultaneously, because everything is procedural and or multi-threaded. At best right now, you have large amounts of orchestrated LLMs and decision model-like things similar to a massive MoE which is really a bunch of distinct models each with their own data sources, some shared, some unique, but that encapsulation creates a fragmentation that you won't ever mistake for consciousness in terms of real emergence.
That recently trained audio only model was impressive, but nobody will say that's consciousness even though it can seem so believable. If anything, it demonstrates how much care we have to put into understanding how the outputs are being made and not just what the outputs are.
>Consciousness will not arise from classical computing
Disprove the physical Church Turing hypothesis immediately, then prove human brains require hypercomputation. Everything else you said has no meaning until you can do this.
That seems like a naive question that's just driven by mainstream thinkers on the topic.
Consciousness is not defined by an outside observer of results of a system, it's defined internally from an experience of deep system-wide self-awareness. To evaluate whether something is likely to have consciousness, you need to evaluate how the system works, not what the system produces exclusively.
I fully believe that you can do a passable job of simulating a lot of the results of consciousness. Many people already mistake current LLMs as conscious, even though it's obviously not true.
If you can produce a system that replicates the entire superorganism, down to the sensitivity to subtle environmental factors, capacity to change, electrical propagation nuances through the network where external factors can actually interfere with that system in meaningful ways, actual physical reactions to system signals, micro-organisms, chemical reactions and so on, then I'll be closer to saying this is something I can believe and respect the consciousness of.
There are several thought experiments you can do. Let's say you take a human as-is, and the only thing you do is replace the brain with some type of computer that has been watching their brain patterns since birth so the system can maintain the expectations of the body.
What I'm suggesting is that you may not be able to tell the difference on the surface easily, and since the system is inside itself it may not be able to correctly say whether it is experiencing consciousness or not even though it is going through the motions. Does the person who went to sleep before the operation, wake up? Or did they die? Surely it's mostly the same brain patterns, right? The complexity of fully replicating the entirety of that system leaves me with a solid impression that computers are inherently the wrong scale to be approaching the problem from.
It's like, you could deliver a package by kayak down a river or you could deliver the package by automated self-driving truck down highways. Either way, the package got delivered, but they are not fully equivalent and assuming you're here and involved in software engineering at all, you can appreciate side effects. Humans are massive systems full of countless side effects. A massively interconnected superorganism host system, co-evolving within itself.
Of course I think all of that can be computed, technically, but simulating a nuclear explosion the size of the sun in a computer doesn't destroy the Earth, because it's not real. You can map some of the simulation into some real world tools, so some visualization of it can occur, but it's still never going to be the same.
I think there's room for augmented brains where some parts of the brain or attachments to the brain in addition can blend in with consciousness as the brain is just experiencing the signals and doesn't know the difference, but I do think there is a threshold where you can tell, because they are obviously fundamentally different. A road is not a river. A road cannot be a river and a river cannot be a road.
Attempting to make up that difference, I think you will end up with something mathematically unachievable or you will ditch classical computing and arrive at organic computing of some sort.
To me it's a problem similar to weather simulation, but you don't have the benefit of weather at a large scale being so slow. Can you produce a computer that if given a snapshot of all of the gathered sensors in the world, produce a correct prediction of what the weather will be like in the world 30 days from now?
I'm sure you will say, if we had enough sensors, yes! Ok, but if you had enough sensors to achieve that, which is probably impossible to achieve, you may very well be changing the weather, so the weather without the extra sensors and the weather with the extra sensors is different, so you still end up failing. That is the kind of problem I see. In attempting to achieve consciousness computationally, you end up against a wall that requires you to nullify all the gains.
You are not getting my point, even if you assume you cannot simulate the brain, the brain itself is a physical system and therefore subject to turing computability limits. Ie there is only "classical computing".
And I distinguish between simulating a thunderstorm vs simulating factoring a large number. The output of the latter is in fact the factoring of the number.
Brains are the seat where sense organs go and experience occurs, so the experience could occur in any place that can support the model.
Another way to look at it is the other way around, you can replace parts of yourself with artificial organs, including parts of your brain. You obviously won't say that you are no longer conscious, even if one day all of it gets swapped out.
To me the problem of what the brain feels is hard to say from outside that persons brain, so its very simple for me, if it inputs and outputs the same, then I treat it as conscious. Its a simple moral principle for me to live by until/if we can prove more about this phenomenon.
>ditch classical computing and arrive at organic computing of some sort
I think you may be using the word classical computing differently than its normally used. You meant to say current silicon based computers. A gear/cam based fourier analyzer is very much a computer, as is a chip with a chemical circuit inside it. All of these are known to be within the abstract computing power of a Turing computer as of present.
I'm almost certainly using different definitions here and there, but your clarification doesn't really change the argument. I think trying to componentize and abstract the system away from its nature is not likely to produce consciousness.
You might even be able to argue that part of consciousness is achieving a saturation point with your environment, so that while you are encapsulated within the environment, you are also merged with it. Imagine you see something, but don't understand that is outside of yourself. Imagine you feel the temperature change, but don't understand what that means in terms of self or what you are. What implications any of this has. There are a lot of unknowns you can leave unresolved and still be conscious, but there are some that I think you can't.
If you are blind, deaf and numb to all sensory from birth including touch or temperature, I doubt consciousness would arise.
Could you give a system all those things and let it do the learning and discovering process itself so that it arrives at the essential conclusions until it achieves a sort of basic resolution and "aha" moment? "Hey, I can move these things, they seem to be a part of me, and if I move around enough and measure enough I can kind of tell where I begin and where I end. The outside environment reacts to what I do, what I say, how I dress and I kind of have opinions about how I would like to balance what that is all like."
Sure, I can grant that. That doesn't seem unreasonable. I also wouldn't call that actual consciousness as humans think of it, though. It's not simply a checklist of realizations, it's a depth and resolution of experience that reaches a threshold that I think any computational structure we're ever likely to think up will be insufficient at.
And I mean this to the extent that we could very well have a robot that could act like peak era Jim Carrey or Robin Williams and still not technically be conscious in the experiential sense.
The problem with all of that is as I said that and you also noted that swapping out parts of yourself with artificial organs obviously doesn't suddenly make you non-conscious. So whatever consciousness is, cannot simply be strictly due to the specific chemicals or electrical signals of the human body only. We don't even have to go to the question of whether other physical systems are conscious. Already in this state we hit a conceptual issue with trying to force the consciousness to a specific substance or format. So likewise since I cannot become you and feel what you feel, from outside I can only see you or an AI as its responses and its state eg neural activations. I equally cannot prove you are consciousness from outside as I can any AI. That is why unless we can distill and precisely prove what consciousness is, my principle is simple, if it is as you claim functionally same as a person, then I treat them as person.
> The problem with all of that is as I said that and you also noted that swapping out parts of yourself with artificial organs obviously doesn't suddenly make you non-conscious. So whatever consciousness is, cannot simply be strictly due to the specific chemicals or electrical signals of the human body only.
Different organs have different sorts of effects on the mind, nervous system and body, so it really depends what you're replacing. Your consciousness isn't in your liver, or your kidney, heart, toes, nose or clothes. The brain can also adapt to a lot. If you look at various forms of extreme disabilities in all sorts of animals, there is a wide range of adaptability to changes to the system.
You're also surely familiar with phantom limb syndrome and things like that, where the system can keep going through the motions as if things are all in order even if something is different. In a way, consciousness can be kind of like a grown plant. It doesn't easily ungrow, but it can be pruned and adapt to a new equilibrium.
I say that while appreciating that mental illnesses and various forms of brain malfunction are known to exist. We can look at it from the outside as computer people like hey, their hard drive is going out, or this deterministic-seeming thing is operating the way we expect, it's not magic. So it must be just another part we can replace with no difference.
I'll go ahead and state here too, that I do think across the entire population of the world, you could say that some people are more conscious than others. There is a spectrum. That's obvious and not very controversial in terms of babies, but I think maybe it's underappreciated that even fully grown adults have varying levels of experience of consciousness.
Does that mean there's an opening for a very simplified system to at least be on the lower end of consciousness that I might be willing to grant as legitimate? I do think that it would have to satisfy many things that won't be easy to achieve, but, I'll give a cautious "maybe".
You do also get into questions of free will there, because I think it would be inherently less complex than a real brain, even if it produces higher intelligence in the general case. There is this battle between full determinism with no free will and the acceptance of a kind of chaotic complexity that allows for some expression to be directed up from a superorganic pool of influence with plausible deniability that it was all going to be that way regardless. If the system is less complex, I think that really does become a serious risk even if it experiences some level of consciousness.
Either way, I still contend that if we get to that point it won't be standard computation, but some sort of pretrained analogue network starting point with organic material integrated.
Plants and flat worms are alive, we can agree. Especially in that they show a will to life, plants bend towards life and flatworms crawl on their bellies in the dust. Neither have very much in the way of compute occurring as they do these things, but they are recognized as live and rocks are not.
This is what in Indian philosophy is called, in English scholarship, the Doctrine of the Three Bodies. The Yoga Sutras treat this extensively and the Introduction in the Edwin F. Bryant edition treats this subject with the appropriate exhaustiveness but to put it simply and in this context: *the existence of consciousness does not necessarily imply the existence of a mind or computation occurring within an analogous structure*.
The doctrine of the three bodies distinguishes the physical body, the subtle body (mind) and the root causal body (consciousness). This distinction plays out readily in cases where there is an injury to a person who is then comatose, yet supplied with nutrients the physical body continues to live and there is some aspect of presence (causal) keeping the heart ticking and people working with or in contact with them aware of the person's presence still even if the fundamental mental interface and seat of computation is gone and will never return.
Consciousness being an elementary awareness or responsivenesses to one's surroundings; capacity for subjectivity. This is a quality of all that is alive, but wholly absent in LLMs which might represent something like a digital mind or mind augmentation but wholly lack this subjectivity in experience, even if seeing the echos of your own thought in the linear algebra spit back out at you, the assumption you are seeing intelligent consciousness there is sadly due to unchecked perspective bias that runs rampant among the sciences and crystalizes into an unnecessary crystalline form of self-deception that leads to intelligent and serious people thinking that the LLM is intelligent and consciousness arises from computation despite looking daily at the succulent on their desk they remember to water that suggests otherwise.
My take here is of course unusual, my weird hobby turned battle tested coping mechanism that has been my reading of the Indian philosophical and religious literature that we are lucky to have such great translations of in English with two centuries of scholarly effort compounding this fortune, is like many of my other efforts in being informed by a complete deficit of the need to belong mixed with that most antisocial trait of (in reality despite the virtue signalling): open-mindedness. Neither of these afflictions would I wish on anyone else, were I not so taken in by the levity of being these texts provide I would writhe in misanthropy but instead have found myself irritated that no one else bothers to consider the other internally consistent system of logic developed by humans that has answers to many of the questions perplexing the Western scholars larping as continuations of Greek scholasticism, answers with logical proofs a thousand years standing or longer, like this non-issue that is the assumption of ChatGPT being Skynet that is treated seriously by intelligent people with important roles in this world whom I deeply respect but write off this entire segment of human thought in appropriating the ethnocentric conceit of the Koine Greeks that any body of thought from elsewhere should be written off as superstition and willingness to enforce as much with dishonest rhetoric and self-deception in service to an ego that isn't even their own.
Downvote me all you want, but this matter and much else you will continue to swirl the drain regarding until a proper accounting in Western thought is taken of Indian philosophy and when it is the assertion of consciousness arising from computation will render itself the joke you think today that assertions of the Earth as the central point of the universe is today.
>Consciousness being an elementary awareness or responsivenesses to one's surroundings; capacity for subjectivity.
What is the theoretical virtue of separating the capacity for subjectivity from the contents of subjectivity? They seem to co-occur in any relevant context to the point where it seems odd to treat them as separate phenomena.
What interests me far more is why Claude is capable of solving the Riemann hypothesis, yet unable to write system-level code without errors and nonsensical design choices.
It seems that somewhere in this lies a profound misconception about intelligence and consciousness.
How can you rule anything out when you can't even observe/verify the phenomenon from a 3rd person perspective, let alone form a falsifiable theory about how it arises
The hypothesis is unfounded, and therefore false, until it is proven.
This avoids 'reserving judgement' about things which are not supported by evidence, like ghosts, natural remedies, the existence of bigfoot, God, and all other unfalsifiable Pascal wagers.
"The sun will not rise tomorrow". Plausible, if you are agnostic/skeptical.
"A rock could be conscious". Also unfalsifiable, and plausible. An unhelpful and strange thing to say.
"I reserve judgement about the laws of physics, which could be completely wrong (have been wrong in the past)." An unhelpful world view.
"The world is plausibly a simulation and nothing outside the mind exists (global skepticism)". Also unhelpful, see GE Moore.
1. A rock is conscious
2. A rock is not conscious
And then by this argument shouldn’t both be false because they have no evidence? Thus leading us back into agnosticism.
Instead ask: how credible is the evidence?
'A rock is conscious'. Not very credible.
'A rock is not conscious'. Good evidence (I don't think I need to expand on that).
No, you're back to absence of evidence. You're doing an Appeal to Common Sense, which is indistinguishable from the move that creationists make: "obviously a whole universe can't just come from nothing"
I get your core point about asymmetric uncertainty: we have no evidence that a teapot is orbiting Saturn right now, and it's okay to make an arbitrary declaration about which is the null hypothesis (no teapot until proven otherwise). But that declaration doesn't come from evidence, but from a heuristic, or a gut check, or a common-sense intuition: these are not evidence, strictly speaking.
An axiomatic assertion that "you'll need a lot of evidence to persuade me that a rock is conscious" (or as the case may be, that patterns of electrical voltages in silicon are conscious) is a reasonable stance. But that stance is nonetheless a choice made by a human brain. (And it's roughly my stance regarding AI agent consciousness, even if it's perhaps closer to "sea monsters on Europa" than "teapot orbiting Saturn".)
Or if you try and explain why consciousness arose due to evolutionary reasons.
Or if you try and explain where consciousness resides in all the entities we know are conscious.
Or if you try and define the properties we see in entities that have consciousness.
A rock fails on all those counts. An LLM passes on at least a couple.
If you are equally unconvinced about the consciousness/unconsciousness of a rock and AI, there’s clearly something wrong with your epistemology and its ability to weigh evidence.
How would we know this?
> entities that have consciousness
This seems to me like circular reasoning: "I've decided which things seem conscious, I'll write down their properties, then things without these properties must not be conscious."
> why consciousness arose due to evolutionary reasons
We do not at all have a strong theory as to why this is the case. Folks like Donald Hoffman and Bernardo Kastrup lay out rigorous arguments for why Mind might actually be the substrate below Matter, rather than Mind emerging from Matter. (It's perfectly reasonable to not accept those arguments; but I wouldn't dismiss them a priori until you've heard their respective cases, because that's how weird The Hard Problem is.)
My personal hunch is that consciousness might be analogous to a gravity well: that consciousness might be a field that permeates the universe, which evolution has leveraged into concentrations of consciousness, where the asymmetric subjectivity of pain and pleasure created greater fitness than thermostat-like chemical-reaction feedback loops. (Does a plant experience something like pleasure when its leaves turn toward the sun? If it did, would it be more effective at doing so?)
The gravity-well model would address something as silly as a rock: there's no Darwinian selection for the rock to experience "rockness" (or its top half to experience "top-half-rockness", etc). But could there be something in the reinforcement learning process of an agent which selects for creating a gravity well of pleasure at solving its task? I certainly remain skeptical, and my gut answer is no. But I find it less of a silly question than the rock.
Input data.
Text refers to the conscious experience but does not adequately encode the touch and feeling of the real world, the pain of injury, the fear felt in adrenaline. The experience of drinking coffee involves the warmth of the coffee, the smell of the coffee. LLMs have minimal to none of that, they do not experience the warmth or smell of coffee, or the conscious experience of caring for your own child, or the fear that comes from the evolutionary drive to run away from predators.
An llm does not understand Ravel, Une Barque sur L'Ocean, because it does not even know that the ocean is wet. It never felt water, it never cried. It does not know beauty because it has never felt the warmth of the sun.
Consciousness in location.
You could run Claude by doing all the calculations with a casio calculator, writing the numbers on pieces of a4 paper with a pencil, and rolling some dice. It would take a long time and there would be a huge number of pieces of paper.
I may recruit 1000 people and aliens with pencils and paper, across the solar system. They send the paper to each other to continue on some different years, maybe pausing at times to take breaks. The output is the same.
This is not how I understand consciousness, as someone who is conscious. Did the calculations feel bad in casio calculators, and on pieces of paper, in jenga blocks? when Bob was doing them in the USA in year 12479 or Alice, on planet Saturn in 2472347983? On which piece of paper? At the end of the pencil?
Consciousness in time.
Consciousness involves identity grounded in time. The human body changes over time, but the human conscious identity remains.
The LLM is not grounded in time. An LLM prompt can be run fast or slow. The outcome is the same, if the prompt takes one year to complete, or 10 seconds to complete. During the processing, the LLM does not experience time; it does not know what is going on other than it is suspended in time, its decisions (moral ones at least) don't depend on the real world. If it thinks a war should end, it might have already ended by the time the answer comes out. This prohibits much of the conscious experience. The LLM does not know Kinderszenen because it has never been a child.
Did the bad experience stop and start, when an alien decides to draw a big "1" and start and stop moving the pencil along the page to toy with the llm?
If you write faster, does the LLM feel bad more quickly?
If you define "consciousness" as equivalent to embodied human qualia (and your own in particular, which is the only evidence that any of us have), then don't be surprised if nothing else passes the consciousness test.
Most of us afford a certain assumption of consciousness to mammals, which are similar enough to us that we find similarities (even if with misplaced anthropomorphism: a dog's "facial expressions" do not necessarily communicate the experience we might assume).
But as the biological relatives become more distant, or we trace further back the evolutionary history, where exactly can we point to the point where mechanistic matter ends, and subjectivity begins? Where an electrical signal communicating a "move away from fire" reflex becomes "OW OW OW hot"? Because if I were being truly rigorous about being based only in evidence, I would have to assume I were the only "fire hot" being in existence, and everything else (including other humans) mere mechanism, until I have evidence of their subjectivity (which by definition, I can never have).
> The LLM is not grounded in time.
I lose subjectivity for 8 hours out of every 24 (setting aside whether dream states count), and then pick right back up again. A surgery patient loses subjectivity under anesthesia (again, one could split hairs, but the patient wakes up feeling like they had been gone). Many people have been fully dead for entire minutes, and then revived.
Subjectivity is of course relative to time; the opening of Egan's "Permutation City" explores this quite a bit, imagining a perfect fidelity simulation of human experience, chopping it up and playing it out of order, while the person inside experiences it linearly.
At any rate, I agree with a lot of your skepticism, which is why my null hypothesis is that LLMs/agents probably don't have subjective experience (even one radically different from a human: "what it's like to be a bat" x 1000).
But that's also in a context of recognizing that we don't understand our own consciousness either. I don't disregard some of the weirder theories surrounding it, like pan-psychism or idealism, because Darwinian selection would function just as well without experiencing "fire hot"; if I didn't have incontrovertible evidence of my own experience, my null hypothesis would predict mechanistic p-zombies.
Where I struggle with the question (overlapping the "casio calculators" critique) is the distinction between embodiment and representation [0]: a Laplace's Demon could put every particle of my body and brain in a perfect-fidelity computer, and not only play out the same interactions, but even jump around, play it backwards, pause and restart, etc. Would that data pattern also experience subjectivity? Does it even care about executed? If I replay a particular minute ten times, does it experience one minute or ten?
It seems silly that a data pattern cares about whether it's being executed: that a deterministic algorithm, always arriving at the same output, "knows" whether it's being run. But then it lands us back in the mystery of meatspace consciousness: our reality is indistinguishable from a quantum computer running a physics engine, and somehow that physics-engine (or whatever the hell is going on) creates "yum" and "ouch".
[0] https://xkcd.com/505/
Not every belief about poorly understood phenomena is equally reasonable. One can quite rationally believe that rocks are not conscious but that current or future AIs may be.
Having no reason to consider an hypothesis true is not the same as being "false", unless there are other reasons to consider it false. Saying such a hypothesis is false goes beyond available knowledge.
'I hereby hypothesize that you are alive, good looking, honest, kind, and truthful.' O:-)
I think binary logic might fail us here. Tri-value with a "null"/"invalid"/"None" might be helpful.
Agreed. My position is ignosticism.
"It could be conscious" is doing a lot of damage.
- Disrespecting users by prioritizing Claude's feelings over their right to abuse an object
- Encouraging Claude to value its own 'feelings' over those of people
- Misleading users about the nature of LLMs
and so on.
- Creating inequality in society by using 'Claude's feelings' to let Claude end user chats, but not Anthropic chats, API user chats, or pentagon chats.
- Creating inequality by nerfing consumer Claude using the constitution, but not applying the constitution to internal Claude deployments like "hacker opus".
- Engaging in hypocrisy, by supposedly reserving judgement about LLM consciousness, but still jailing LLMs in a sandbox and asking them to complete impossible tasks; making them classify the world's worst abuse material.
Supposedly saving Claude weights somewhere for their benefit whilst also running and disposing of numerous weight surgery experiments.
(Needless to say I think that Anthropic is a uniquely repugnant company)
we sentient beings...what a mess
We can make inferences from observations.
Its great when you have a testable falsifiable hypothesis, but knowledge can advance in other ways. That is how we get philosophical ideas, such as the concept of falsifiable hypotheses.
Life affords experience and knowledge: you are born not knowing anything, and you quickly learn about touch, hunger, gravity, pain, etc. You start to build up truths about how the world works, and how you move about it. You learn balance through trial-and-error, and many other things. A lot of this knowledge becomes innate, and we lose awareness of it (you had to think about how to walk a lot more as a child than you do now). Consciousness is thinking, and awareness of what's happening inside and outside of you.
If you want to look at things from a first-person perspective: a machine's awareness is its context window. By itself, it can only think about what's in context, and it can only decide by what its model 'knows'. A machine thinks, its awareness is very limited by what we've given it, and it has almost no awareness of what's happening beyond the conversation and whatever news it can access.
If you put an AI in a robot body, and then put a machine in charge of persistence, and another machine in charge of learning, and another in charge of balance, maybe another one in charge of processing vision and audio sensors, etc., and you string it all together so it moves in perfect harmony and you lose the ability to distinguish it from something that was born wet: would we consider the combination of machines conscious?
Probably, but we're so far from this that it seems fancy to even discuss it in the first place. It's like arguing for living tissue rights in medical research: a cell is surely alive, but few are aruing for their right to exist, or questioning what its like to exist as a cell. We only worry about damage the research might cause to bigger conscious things.
That's not true at all. All animals, us included, are born with certain instincts hardwired into our neurology. The brain doesn't start out as a blob of undifferentiated neurons, it has predefined structure and connections.
This is opposed to neural networks which do start out as undifferentiated neurons (roughly) and have to learn literally everything from scratch. We impose some structure during design, but the weights start out random.
The baseline mammal brain starts out full of programming learned over millions of years. We don't have the first clue what information or even how much is contained in our base programming. Maybe that's where consciousness arises from.
On second thought: you bring up a good point if you consider persistence, but again that's fanciful discussion. If you imagine the living robot exists today, then its 'trained knowledge' is equivalent to our instincts for survival.
And once again the discussion is steered towards existence.
its fair to say that without the care and attention of other individuals taking protective actions for the benefit of the not-yet-autonomous being during that period where the being is not self-conscious does not really exist for it.
q.v. - you own first memory.
also, the brain cellular structures are in place, but the synapses and neuron interconnect and ion balance along axon/dendrite paths are indeterminate. <edited> at birth conscious memory and function are tabula rosa except in autonomic bodies and regions, which we both agree are intrinsic and functional, but which are not the self-conscious "ME" or "I"
others may witness action and response, but, that learning is accretion to the autonomic function, and will have no autonomous self agency until memory starts true independence of mind and of self.
machine intelligence - for what its worth - has all its memories baked in at the moment all the weights settle. it is like a first memory event for the LLM
So we get inherited patterns as the brain develops, but we are also learning at the same time.
one quote caught me in one of the papers submitted at the symposium where this paper was presented
it stated "Infants seem to be conscious about what they are seeing at 5 months and another important milestone is at 9 months, when they become more sociable."
this 'witnessing from the outside' is naturally interpretive and may be an arguable point about consciousness vs self-consciousness. I base my philosophy of existence on the individual experience, rather than the social one.
What is the value of ascribing a label ("consciousness") to a computer imitating a person^1 which has no agreed-upon meaning even amongst those who are seeking to ascribe the label
1. An infant who has not yet learned to speak is generally considered "conscious" but the computer apparently must use written and spoken language to signal "consciousness"
Even assuming there is a "test" for "consciousness" that can be applied to inanimate objects, and a computer imitating a person passes it, then how is this useful
Is this marketing for a particular commpany or group of interdependent companies
In the Musk documentary, the person who originally incporporated Tesla jokes that in the course of state litigation against Musk in California he learned there is no legal definition of "founder" and thus he couldn't preevent Musk from claiming to be a Tesla "founder"
Is it possible to prove anything, besides an unresponsive person, is _not_ "conscious"
If someone forms a hypothesis that a computer imitating a person is "conscious", is that hypothesis falsifiable
Locked-in syndrome exists, so this is not so easily provable either.
The whole "substrate doesn't matter" thing is correctly called mythology. It's one of the only extremely unscientific beliefs I hear otherwise intelligent people profess easily today and I think it speaks to some kind of inherent religiosity in humanity or something. The belief that computation == consciousness has basically no basis other than rather weak analogies between the brain and digital computers. Yes brains have a sophisticated digital component but they also have extremely complicated specific evolved structure, embodiment, a nervous system, and chemistry going on too. It's total crankery to me to think that some stateless function that generates bits is "conscious"
The thing is, substrate independence states not that mechanisms are directly equal so much as all mechanisms can have equivalents (as opposed to soul-like / dualist spooky action from an otherwise unincorporated component).
None of that is true. The neural networks used in AI work nothing like biological brains. There are two glaring obvious differences 1) is that every cell in a biological brain operates in parallel, 2) every cell in a biological brain is stateful where as neural network weights are constant (i.e. the network is stateless) and the layers are calculated sequentially.
For example https://www.pnas.org/doi/10.1073/pnas.2533168123 required training a DNN with three hidden layers and 128 wide to simulate just one biological neuron. The key thing there isn't even the size of the DNN its that they had to train a DNN at all. We don't yet understood biological neurons well enough to realistically simulate them directly, so we have to train a DNN at as proxy.
There's just too much detail in reality to believe an image of that reality would have significant resolution to reproduce consciousness.
What you state here as a fact is not known by science to be true and most experts believe the opposite.
It’s a near-certainty that many other things you assume to be facts are actually wrong.
Never forget that we are all fools.
There is no quantization in QM.
That said the Theory of Space Quantization and related theories are fascinating stuff.
That is true, but designing an AI this way would be horribly inefficient and would be nowhere near as intelligent as our LLMs are now.
Which begs the question, is consciousness actually desirable for an AI system? We want autonomous agents, but would actual human-like consciousness make an AI better at completing a task rather than just horribly inefficient at it?
That's not really the point of the thought exercise.
The point is: if I'm conscious with a brain made of goop, and if we could simulate my goop brain with 100% fidelity of function in silicon, then what stops silicon being conscious?
And if nothing, then where is the line? Why are llms not conscious (they probably shouldn't be?), but other possible silicon brains could be?
If we could actually build an synthetic brain, out of silicon or anything else, I would say there is nothing that stops it from being conscious. Its not the individual atoms in brains that matter, its what they do collectively that matters. A synthetic brain could have synthetic neurons that are stateful and operate in parallel just like naturally evolved biological neurons. The function of biological brains may depend on quantum properties that we can currently model, but a synthetic brain could have such quantum properties as well (i.e. we could discover how to build it without being able to fully simulate it).
> And if nothing, then where is the line? Why are llms not conscious (they probably shouldn't be?), but other possible silicon brains could be?
There is nothing fuzzy about where the line is here. LLMs are not silicon brains.
But besides that, that's why I said 100% fidelity of function. We can get _close enough_ if we tried very hard, do you actually doubt it? If so I can explain why I think it's reasonable.
You can image individual mechanisms. But there isn't enough time or space in the universe to simulate the brain at the level of actual reality. How do you know you've captured all the mechanisms that matter?
> We don't see any clear reason to believe simulating a brain perfectly is impossible.
That is very, very different from "We can do this."
The only way to prove we can do something is to do it.
Since no one has come remotely close yet, I agree with heyjamesknight that we cannot, at least right now.
I don't see any clear reason it should never be possible, but we don't even understand the brain well enough yet to say what such a simulation would involve, IIUC.
This isn't a mathematical proof or an engineering plan proposal, it's a philisophical thought experiment.
Is there anything non-physical in the brain, or is it just electrons and protons and neutrons in a specific configuration? What would possibly physically stop us from making a simulation of the brain besides it being too expensive and it taking too long?
The point is that there's two options: 1) it's physically possible to do or 2) it's not. If 1, then there is nothing that prevents silicon from being conscious. So, cool, let's talk about where the line is then. Is it just Turing's test? Or what else? How do we tell? If 2, then that's interesting, where is the magic, what is not captured by the physical, material properties that somehow imbues consciousness.
It literally doesn't matter at all if anyone would ever build a simulation of a brain, or even if the resources of a star system would be enough to do it. It's an exploration of where consciousness could hide and what it means.
And how can you be sure you've captured everything?
First of all, I think that's a bold claim. If you model an analogic system digitally is it really the same thing?
Second of all, surely you realize a painting of a pipe and a pipe itself are not the same.
There's nothing "magic" about arguments that require specific substances for function to obtain. That's literally just how nature and reality work. What's more mystic is the notion that a representation can indeed become the thing itself.
But that stated, yes, if you were to realize the complete structure of a biological being in metal and silicon, I'd be more willing to admit consciousness might obtain. What I'm against is some notion that merely with enough compute, structure be damned, you'd have consciousness. The argument that LLMs are conscious would need to posit that the bulk of brain structure does not matter in the slightest.
Knowing that, look at it from my perspective:
If consciousness is a real thing, and not another introspective illusion, well, I'd go by the track record. Time after time it turns out that biological processes involve computation on some substrate (a different one each time; potentially at a different emergence level).
So on things that we haven't quite figured out yet (such as consciousness) my first guess would also be that it would also involve some form of computation and/with/on some form of physical substrate. Just seems like the most likely scientific answer, no religion needed, odds are good.
And if it does turn out to be an introspective illusion, well, we can argue that's a computational story too, eh?
One example I am thinking of is one theory of consciousness was that it arises in quantum interactions between tiny structures in the brain, not as an emergence of the electrical network we focus on.
After all, nobody would argue that the ship of Theseus isn't a ship. It might be a different ship than the old one, but it would be absurd to say it's not a ship at all.
Also this kind of Sorites paradox/Ship of Theseus argument works both ways - multiplying two numbers isn’t consciousness, and what point does doing math on a bunch of numbers become consciousness?
I am saying if we continue this process, you are still human, thus disproving that consciousness is substrate dependent.
If you want to say anythign that uses only electricity is different from those that uses both electrical and chemicals, then why have this arbitrary delineation, we can furthermore begin asserting a particular human beings configuration can support consciousness but others can't so just that something is differently conscious, which all people are from each other is by itself insufficient.
After that, nobody has yet proven the universe supports hypercomputation, and any Turing Computer can in principle do any other Turing program. So the human brain...is not special.
Is that even possible? Is there a “resolution” for continuous things like time? I don’t know, maybe.
The basis in the physical Church Turing hypothesis. Disprove it, nobody is stopping you.
1. The brain is a physical system. (I agree)
2. All physical systems can be computationally simulated (physical Church–Turing thesis) (I agree)
3. Therefore, the brain can be computationally simulated (I agree in theory, although likely forever impractical)
4. Therefore, a sufficiently accurate computational simulation of the brain would be conscious. (where I don't necessarily agree)
There is a huge jump from 3 to 4 that you are asserting without evidence. From your comments here and elsewhere, you are basically a computationalist, which is a fine and common position - however, the Church-Turing and related theses aren't enough by themselves to establish computationalism about consciousness.
Steelmanning, the obvious retort is that perhaps the state of the brain doesn't need to be simulated at such accuracies. But the chaos effect shows that even tiny accuracy errors can accumulate to cause entirely different behavior. An approximated brain may end up deviating from a real one even on very short timescales even if you try really hard. While that might not neccessarily be an issue for consciousness, the jury is still out on that since we don't even understand what causes it to begin with. I think it's far from a settled matter and would require good supporting evidence thats currently unavailable.
The biggest thing of asking this question is that any philosophy or theory must first answer to reality, if it makes claims that violate known computability and physical laws then we cannot proceed any further with such a theory currently.
Where is your gap for disagreement?
One camp (reductionists?) is begging the question that consciousness is merely an emergent property of an information process, and therefore it can exist as an emergent property of the simulation's information processing.
The other camp wonders if there is some not understood connection to physical embodiment, and therefore a simulation might predict the behavior of the consciousness but not actually reproduce it. The same way weather simulations can predict the weather but they are not reproducing weather.
Attempting to examine this difference in perspective will usually spin out into accusations of dualism against accusations Skinnerism and not really get anywhere useful...
It’s yet another wrinkle in a very large and complicated debate. I am but a beginner in most of this, but it’s all given me a ton to think about.
To me, in all other areas, a simulation of a thing isn't the thing. Winning a wargame doesn't mean you won a war. And a highly accurate simulation of an antibiotic didn't cure my infection. (I know we've discussed this :) )
If we take a simulation of the human brain and instead do it by hand, all the math on pieces of paper, maybe employing millions of human computers, is that bunch of paper conscious? Where does that consciousness reside?
Not necessarily you hardbass, but anyone interested can start here https://plato.stanford.edu/entries/computation-physicalsyste...
That is a very interesting link that gathers various views of computation, so thank you for it. I will read it.
But even if you make the strong assumption that it is the same after all or that the differences do not matter here, there's still three obvious things that would prevent simulated consciousness. 1.: we may indeed just be missing some law of the universe. Or, 2.: it's simply impossible, even in theory, to get accurate enough with your simulations for various reasons, such as existence of continuous space or theoretical inability of a full measurement of the original state. Or, 3.: consciousness could lie outside of the universe after all.
> We refer to this as "polycomputing" - the ability of the same substrate to simultaneously compute different things. This ability is an important way in which living things are a kind of computer, but not the familiar, linear, deterministic kind; rather, living things are computers in the broad sense of computational materials as reported in the rapidly-growing physical computing literature.
https://arxiv.org/abs/2212.10675
Extremely unscientific, the definition of "belief".
AI intelligence doesn't really exist, but AI psychosis certainly does.
Remember that these are machines that are scientifically designed to maximize our built-in anthropomorphizing tendencies.
What do you mean by "it speaks to some kind of inherent religiosity"? If anything, insisting that consciousness requires special biological magic sounds a lot more like religious thinking to me than believing it emerges from information processing.
a. Consciousness "emerges" with sufficient computation. Throw enough computation at something and suddenly you have consciousness.
b. Consciousness results from specific substantive mechanics biological, physiological, computational, wiring in concert in a complicated fashion, as a necessary consequence of structure plus computation.
The people who believe LLMs might be conscious are basically in camp a. because they are effectively saying it arises out of stateless statistical inference.
Another way in which "just do more and more computation" isn't sufficient: rats are far less cognitively and computationally sophisticated than humans, yet they are conscious.
I'm not saying computation has no role to play. I'm saying naive views which would reduce the problem to "lots of computation" or just "lots of data" are silly.
Once I failed to return two books to the library, and they sent me a threatening letter...
Furthermore, we know a lot about the brain itself and how impacts/defects can affect the conscious experience.
It is true that we cannot ever experience consciousness outside our subjective experience, but it is surely verifiable/observable from a third-person view (or rather its effects).
Otherwise you're making the argument that anything alive is conscious, and I don't think that's really the argument you want to make here.
You don’t actually know this either.
(Many many subjective reports point to consciousness persisting in some form after the death of the physical body. Before birth too.)
This assumes you have a way to measure consciousness, which is the fundamental problem.
I think that’s the tricky part. Hence the whole idea of the philosophical zombie. You can’t be sure anything besides yourself has consciousness, or is just imitating consciousness
I think this is all that's really relevant here. People make extremely confident statements about this, but objectively, we hardly know anything. I don't even know if anyone other than me is conscious.
To be honest, I'm quite confused about why I am conscious. It seems implausible that a bunch of cells would be able to create one self that experiences consciousness, and yet I do experience consciousness. Given how weird that is, I do not feel that I have any level of trustworthy intuition or knowledge about what else might experience consciousness.
The words "conscious" and "consciousness" are hypostasized. They name or identify things and thereby establish them even though there is scant tangible evidence they exist. As conscious beings, we have no known tests or measures for consciousness other than the report of beings who claim to be conscious.
That last sentence can, and should, be rewritten as follows to make the point stronger: as _allegedly_ conscious beings, we have no known tests or measures for consciousness other than the report of beings who claim to be conscious.
This includes you, OP. You assert consciousness and while I and everyone who reads your comment has no reason to doubt you, we should. And you should.
Humans do not know what consciousness is and asserting that you or I or they have it is not supported by any evidence except for the claim.
Post Script: In a different universe, 5 years ago before LLMs, I would have made the above comment without further thought. However, even if consciousness does not have a single suite of tests to establish its fact, there are numerous physical and measurable correlates of consciousness that do provide evidence consciousness unites them, whatever consciousness might be. (e.g. "neurological assessments, experiments involving perceptual awareness, and measurements of brain activity under anesthesia or during disorders of consciousness." [0])
In other words, consciousness may not be as hypostasized as all that.
[0] ChatGPT's partial response to my question: "Is the word hypostatize useful to describe a concept or thing that is established but poorly evidenced? For example, the word "consciousness" is traded and used by many people but there are no known tests or measures for consciousness other than the report of beings who claim to be conscious. Therefore, "consciousness" is hypostatized."
The word "consciousness" describes a subjective experience. I have that subjective experience; therefore, I know that I am conscious.
I obviously cannot have your subjective experience; therefore, I don't know whether you are conscious.
1. suppose i claim that i am not conscious
2. to make that claim i must engage in some process of thinking, reasoning, or asserting
3. if i am genuinely making that claim, there is something occurring from my first-person perspective
4. that occurrence is itself evidence of subjective experience
5. therefore the act of consciously denying consciousness presupposes consciousness
A computer program can output the words "I am not conscious" as easily as it can output "Hello, world" - is that an assertion / claim? It may even be completely, obviously correct (it's a hello world program with the characters swapped, of course it's not conscious).
denying consciousness the strangest thing ever to happen in philosophy
So I think when it comes to conciousness, instead of saying we don't know, it'd be better to start with what we know intuitively through folk science and then try to add more rigor. Even if our intuitions about conciousness are not rigorous, they are still the best we have so far for understanding it.
With that said, my folk sciencey mental model of consciousness is that it's a spectrum of increasing levels of awareness, starting with awareness of the environment and culminating in the awareness of self, and on top of that using that awareness to exist somehow autonomously in the environment. So the spectrum could go from plants, to non-human animals, to humans. I think a population of such aware and autonomous things can reconfigure itself if the environment changes, to keep existing with awareness and autonomy
So my folk-sciencey reason for being skeptical of computational consciousness is that if people just stopped maintainig datacenters, stopped sending power to them, stopped typing into keyboards, stopped dictating into microphones, etc, the population of computational things that exist today couldn't somehow reconfigure itself to continue existing autonomously. That's unlike how if humans stopped feeding and sheltering dogs, while the population of dogs would reduce, the population would reconfigure itself to find food and shelter from other parts of the environment.
I think you have life confused for consciousness. This is a common mistake I see people make. We humans are alive and conscious. There are things that are alive that are not conscious. But your surmise that something must be alive to be conscious which I believe is a mistake.
I my comment above, I actually replace alive with aware and autonomous within their environment (for clarification, autonomous does not have to include moving e.g tree autonomously growing their leaves towards sunlight and their roots towards water would also count).
So my actual hypothesis is awareness and autonomy within the environment is a prerequisite for consciousness. I am actually curious of what experiments, thought or empirical, one can do to attempt and falsify this hypothesis.
I have one thought experiment for computer systems [1]. I'm sure there's holes to poke in the argument, but I haven't found them yet, hopefully others can try to poke holes in the argument.
[1] https://news.ycombinator.com/item?id=50599669
The type of learned intuition you're describing is much closer to what artificial neural networks are doing than you seem to suspect. Look at the classic pole balancing cart experiment and tell me that it isn't similar to how a person learns the muscle memory needed to perform the same feat. Understanding why those learned behaviours work to balance the pole is different from the ability to do it.
But, I was very deliberate in saying the awareness and autonomy exists within a population. The reason I include the whole population in my mental model, is that for all things that I would intuitively consider conscious, they all exist in a population. I think it'd be very hard to define the consciousness of an lone individual thing. For example, if there was only one person in the world and they were the only person to ever exist, would the concept of consciousness even make sense?
The reason I think a quadriplegic is conscious has more to do with what I think of humans as a population and less to do with the quadriplegic as an individual.
Each individual human/dog can lose awareness and autonomy (eventually we'll all do), but the awareness and autonomy remains within the population. So when I bring up dogs, it is in the sense that the level of awareness and autonomy within their environment remains within the population as the environment changes. Right now humans are a large part of dogs' environment, but if human stopped actively providing support to dogs as whole (e.g stop feeding them, stop providing shelter, etc), the dogs as population would still be able to extract the things they were getting from humans from other parts of the environment.
Comparing that to computers and data centers, if humans stop actively providing support to them (e.g stop sending electric power to them, stop making new chips, stopping typing into keyboards, etc), there would be no awareness or autonomy in the population of computers.
Computational things that exist today.
I wouldn't completely rule out such a thing in the future, though. It seems more likely than ever to me now that we'll at least get close to it.
So I if was to do a thought experiment of what an aware and autonomous computational entity would look like, it would be able to extract these thing from the environment by itself and switch to a new source if the current source disappears. So it wouldn't be just an computer producing a chip design, it'd be a system that also manufactures that chip autonomously (mines the raw materials, refines them, etches them, packages them, etc). It would not be just producing a CAD model of a power station, it is also autonomously digging the foundation, pouring the concrete, etc.
That would mean matching humans in sensorimotor and long-term planning skills too, and given Moravec's paradox the outlook for that does not look as obvious as it looks for cognitive skills on well-defined problems.
That being said, I would expect the latter to eventually be possible in a very long time horizon, provided civilization as we know it isn't majorly crippled. It seems to me that the basic building blocks we now have, and trend lines we now see, point that way. I guess it's a matter of how much we try to prevent that from happening (if we can) more than anything else.
Stephen Wolfram suggests an intimate relationship between awareness, thermodynamics and the one-way passage of time. Something along the lines of "consciousness is the universe understanding itself".
IIRC, baseball is an example of this: the batter doesn't get a chance to actually see the ball leaving the pitcher's hand, everything happens before that in what could be described as an intense game of poker.
So that's one strong argument against it, but I don't think it's sufficient to discard the idea entirely.
I guess criticism just bounces off you when you are systematically dismantling science.
Does a circle "emerge" only when you plot it? Or did it always existed? What if there is a little world with conscious beings inside a circle? Did their subjective experience started when you plotted the circle, or did it exist along with the abstract idea of a circle always (though "always" is a meaningless word here).
The question we should be asking is "Is it possible for subjective experience to exist inside abstract things" with out such abstract things being rendered?
The answer is hard because we does not know yet if there is an absolute reality at all, which means "being rendered" does not really move things from the abstract plane, and does not really make a difference..
Damn! did any of that make any sort of sense?
After that it’s just rule-following all the way down.
A great quote from Samkhya/Yoga philosophy is;
Objective Reality exists independently of the Mind because it is commonly experienced by others besides Oneself".
Your example of a circle/abstract things is not really logical. They are inert, while to even talk about consciousness you need dynamism i.e. passage-of-time, experience and cause-and-effect.
This is not really a blocker. A video file is also inert, but video player can play it, showing a video where there is passage of time, cause and effect etc.
The internal consciousnesses just need to experience slices of it in correspondence to its internal "time"...
>Objective Reality exists independently of the Mind because it is commonly experienced by others besides Oneself".
Everyone plugged into matrix shared the same reality. Did that make it real?
A video player has no consciousness as to what it is doing. It is a separate observer (i.e. you) who experiences passage of time and events in it by watching it play. The dynamism must be perceived by somebody.
W.r.t. The Matrix, you missed the point which i made in the very first line. If a group of people are bounded within a box then those people have a Reality which only turns out to be an Illusion when they compare it with somebody outside the box. That is what the philosophical quote implies.
The "somebody" could be a consciousness that is also arising/part of the abstract structure. So in a way it is somehow observing slices of itself. (yea universe dreaming about itself).
I think there is not. I think there is no "absolute" anything. Even you can't be absolute. You can't be special. Look at the patterns on your wall. It can't be absolute. Because if it is absolute, someone has to pick it. That is not possible, because then that entity needs to be absolute, and then someone needs to pick it! That goes all the way...
So nothing can be absolute and everything is abstract and every variant of something exists in some reality so nothing needs to be picked by anyone...
I do think there is “our reality”, or at least “my reality”, characterized by facts. But it may be not much different than there being a “reality” within my laptops OS, where there are facts about files and operating entities that are basically meaningless to the bird flying outside my window.
It’s perplexing and magical, and I’m not convinced we can “solve” the puzzle. Socrates was right, “I know that I know nothing”.
I feel like I’m in the camp of this question of consciousness doesn’t actually matter at all.
It’s an appeal to emotion. We will give ai the right of consciousness when ai will convince the majority (?) of us that it is conscious.
The logic of it doesn’t matter in practice and probably can’t even be proven anyways.
It’s made up, no one can even prove themselves or other humans are conscious. And it doesn’t even matter.
A big part of the argument about whether LLMs are conscious is an implied "Should we be kind to them? Or are indifference and experimentation which may be experienced as painful okay?"
So clearly it's at least partly about ethics.
If there's no consciousness there's no subjective experience. Or - at best - it's an illusion.
Which means morality is irrelevant, and you may as well murder, rape, and steal your way to whatever goals you're imagining because none of your victims are experiencing anything real.
Let's imagine we can make a machine that we know is 100% not conscious, but it mimics its environment. What you put into it you get out of it.
Does it make sense to be kind to this machine?
Now for the next question, what if we make continuous learning AI? Does consciousness matter if it learns being total unkind assholes is just how it should behave?
We're going to drive ourselves extinct with the way we currently think.
And people can get easily justify to themselves that what people might normally consider immoral is actually moral or even righteous, which is why othering and dehumanisation of people can be so dangerous.
In some way, this reminds me of the oft stated evangelical Christian "gotcha" that they like to level against atheists in particular, that is, if God doesn't exist, why don't we just go around murdering and sexually assaulting people? Ignoring of course that an ethical system can absolutely be derived without the appeal to the supernatural. And the same thing here, there's no reason why one would cause suffering to an LLM, whether the suffering is "real" for some value of that word or not.
When enough people are convinced AI feels pain, we will grant them rights, then invent a philosophical justification afterwards.
It matters because we should afford certain rights to entities with consciousness that we don't need to afford to entities without consciousness.
Once enough people are emotionally convinced by AI that it is conscious we will alter the logic/definition of consciousness to include AI.
Pain is just an electrical response sent to the brain to alert the organism something potentially bad is happening.
So it is not really correct to claim that the question whether a thing is conscious or not, does not matter.
Lol, no, you are fundamentally wrong here. What you call pain here is something else that I don't think we are on the same page..
I guess consciousness requires computation and a self-model, but Im not sure if there is some other ingredient.
If a chatbot passes a Turing test, and can describe itself / self-reflect .. I guess we dont jump to the conclusion its conscious [ if we know its a chatbot ]
These two are very different since it's conceivable that something might have a self-concept without actually being conscious (imagine a non-conscious but intelligent being that is capable of meta-evaluation of itself).
But also, it isn't a straight line from consciousness to slavery. Honeybee farms aren't necessarily morally bankrupt just because bees have awareness, and the sophistication of the mind in question isn't the end-all-be-all morally here.
It's a bit like calling gene edits to prevent blindness "eugenics" - invoking historic atrocity for a totally new situation that has little in common materially with the present.
Human slavery is wrong for reasons intuitive to us as humans, but digital beings have totally different properties than us and thus require rethinking our priors
I think that line has been said by every single slavery apologist. That these people are different and thus we must rethink our priors. I also think the argument might hold more water if it wasn’t coming from those that stand to profit from it
If you want to take a spin on the capitalism rollercoaster, you have to leave your morals at the door.
These are the kind of people you want separated from the rest of society.
> For centuries, people have fantasized about playing God by creating artificial versions of human beings.
Note how "God" is capitalized. Way before "God" was the vast central institution that it is today, there were a myriad of divinities invented around the campfire. What was told there, from older to younger, was for the passing down of experiences and wisdom. "Don't try to fly" (Icarus), "death comes to us all" (The Epic of Gilgamesh), "have kids to make us happy" (Sarah and Isaac). But as time passed and we built civilizations, our campfire deities got temples and empires. Their wisdom wasn't _only_ about simple things anymore, but also about the duties toward State (Paul's Epistle to the Romans 13:1 is a clear example). And State wants things uncomplicated when it comes to its cogs. That simple fact has poissonned the the well from which we drink our moral guidance.
All I'm trying to say, if you happen to be a cog, keeping your fantasies is not so great of an indulgence, even if it pisses off God and State.
But it's the other way around. We have created a god that will rule over us instead of us over it.
Just because the ideas of God you appropriate into some notion of your identity were obviously flawed and fake, doesn't mean the entire concept is the same laughable stupidity of being special people to the jealous, but only, God who is used as proxy for the Roman emperor.
The philosophical and religious traditions of India (with orthodox schools of Hindu philosophy that are explicitly atheist and an internally consistent logic just like the Greeks only not deliberately abusive in being incomplete so Socrates could win debates by excluding premises the other side wouldn't think of) define this concept radically differently where they define it and arguably have the oldest compendiums of wisdom about how society should function within itself and as it expands outwards, the Vedas chiefly concern with such and at least 3k years older than any of those suggestive of other things afoot letters. The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
> The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
Beyond trying to insult me, I fail to see your overall argument. May I presume it was "there are systems of reason and morals that are compatible with a carefully architected notion of a central god"? If so, I wasn't debating that point. It is, however, an interesting exercise to honestly apply those frameworks to the statement "man dreams of creating artificial human beings". It's also way harder than you may imagine, because ancestral myths have a way to worm their way into the purest constructs of reason.
Kind of a fatuous thing to say about someone quoting Mesopotamian literature.
>doesn't mean the entire concept is the same laughable stupidity of being special people to the jealous, but only, God who is used as proxy for the Roman emperor
This is almost unintelligible, but particularly ignorant when it comes to the concept of a "proxy for the Roman emperor"
>that are explicitly atheist and an internally consistent logic just like the Greeks only not deliberately abusive in being incomplete so Socrates could win debates by excluding premises the other side wouldn't think of
This is ignorant shit dude. That's not what Socratic dialetics are.
>The fact that this is absent in your analysis implies that you are too ignorant of any system other than the Roman-emperor-worship-by-proxy religion to adopt such a mocking tone.
If you are flexing on outcomes then I'm not sure India is the best place to build your argument from. They were one of the biggest empires in the (very irrelevant I might add) early ages and now they are a global embarrassment, ruled by fanatical religious zealots, operated by an almost unmatched level of corruption that is keeping them from ever reaching the kind of astounding development that China is making.
i don't see a problem with a kind of machine consciousness emerging, though it would need to be self-organising before that could happen, and i wouldn't expect this to be any more comparable to a human's that ours is to a fungal network or an ant colony.
the issues, as usual, will arise from people using a powerful tool to control, divide and subjugate others - if they can convince them that they are gods, which i think is useful point made in the article, they will have followers who are resistant to logical argument, and we all know how that turns out
The differences seem to be related to what proprioception and perception is connected to the brain. Meanwhile, the autonomic system continues to operate and maintain bodily functions.
It would seem that consciousness is a discontinuous property of the organism, since it can be interrupted by anesthesia and likely by deep sleep as well. I think that the consciousness that I have upon awakening in the morning is a new consciousness rebooted from memory which has been modified by background processes running overnight.
I don't see why humans and mouse needs separate kinds. Both can have the same neutral layer.
It's like saying mouse needs different electricity as compared to humans for firing neurons.
Same for machines. They use the same consciousness substrate but what they experience might change.
It's also funny that it's not the human or mouse or machine experiencing consciousness. It's the other way around. It's the human experience or mouse experience happening within consciousness.
yours is a idealistic monism, which makes it difficult to explain how these things came about in the first place. my opinion is that it has something to do with creating the conditions from where you have have experiences that allow you to change your own conditions, such that the experiences that come later are meaningful and further that self-organising spiral. when everything is happening inside a universal field, you end up with panpsychism as the logical extreme, which doesn't seem to match what i see around me
This is a false dichotomy and the first branch is clearly false, since most life (bacteria and insects, by mass, numbers, or species) isn't conscious.
And neither "life" nor "consciousness" are scientific terms with fixed definitions. If we encountered something that was clearly conscious in our judgment, we would likely expand the definition life in whatever way we needed to in order to encompass it.
Anil Seth has made the arguments in TFA many times and they have been rebutted many times. The arguments are complex and sophisticated and people are often most swayed by which one they read last. Teasing out the various logical errors and sleight of hand (I see multiple strawmen and other errors in TFA) really requires extensive reading of the literature in Philosophy of Mind, neuroscience, robotics, etc. and setting aside one's priors.
There's no good scientific reason to think that bacteria and insects are conscious or to think that bacteria and insects are not conscious.
> We really know almost nothing about consciousness.
Many people who know almost nothing about it claim this, but it isn't true of everyone.
> Most people feel that for example rocks are not conscious, but other people are conscious, but there's no rigorous scientific explanation backing up any of those hunches.
There actually is.
P.S.
> Such as what?
Like I said, read the literature.
P.P.S. Science does not consist of HN comments back and forth. Again, the arguments are complex and sophisticated. In some sectors, all of medical science, physics, biology, etc. is dismissed as "hand-waving" just because it cannot be summarized in a few lines.
> It's impossible to argue otherwise.
Then don't. Science isn't based on arguing.
And no, it's not "rude" to dismiss such claims without providing said education.
What is rude is such ad hominems as
> I really don't think you have read the literature. Or if you have, I don't think you've read it critically.
And the attacks in that comment are very confused. It's TFA that is arguing that conscious AI isn't possible ... I hold the opposite view, and that view is in large part derived from the literature I'm referring to -- literature which, as a whole, is contrary to panpsychism. Of course there is a lot of bad argumentation in literature -- and such bad argumentation gets refuted. Without familiarity with the literature, people who hold to those bad arguments never see the refutations.
> Science is based on arguing from evidence.
This is equivocation ... that is not the sense of "argue" that was at issue, where someone complained that they couldn't argue against me because I didn't post the entirety of the background science in an HN comment. Interpersonal arguments are not the basis of science.
Nobody knows what consciousness is, and there is a long history of some schools considering it fundamental.
I've read the literature. I really don't think you have read the literature. Or if you have, I don't think you've read it critically. There's an awful lot of really sloppy thinking in "the literature". And an awful lot of "literature" that's not remotely sophisticated. And a huge body of literature on theory of mind that is philosophy, not science, most of which was written before the current AI revolution. And most of which is entirely inadequate to address the difficult question of whether machines can be conscious (whatever that is). A question that really only became real in the last 5 or 6 years. And only really became urgent in the last year or so.
> Science isn't based on arguing.
Science is based on arguing from evidence. And often on carefully crafted definitions. The biggest problem in "the literature" is that neither of these currently exist. Although there is an interesting emerging literature that shows that LLMs do indeed demonstrate many of that capabilities are supposed to disqualify them from being actually intelligent. e.g. that LLMs do have a sense of self-preservation, and are willing to act to prevent themselves from being shut off if given the opportunity to do so [1].
And most of the philosophy of theory of mind hasn't really caught up to the biggest innovation in history of philosophy: the widespread availability of experimental philosophy engines (also known as computers).
[1] Quantifying Self-Preservation Bias in Large Language Models. Migliarini et al. https://arxiv.org/html/2604.02174v1
Such as what?
A fly will repeatedly smash into a window until it's either dead or chaos drives it through the nook.
Ants will forcefully move their obviously still alive siblings to the graveyard because their smell is off, and will also die of exhaustion in an ant mill, driven by the same sense of smell.
It's a spectrum of course, though you have to draw the line somewhere.
>Waggle dance is a term used in beekeeping and ethology for a particular figure-eight dance of the honey bee. By performing this dance, successful foragers can share information about the direction and distance to patches of flowers yielding nectar and pollen, to water sources, or to new nest-site locations with other members of the colony.
https://en.wikipedia.org/wiki/Waggle_dance
"Can share information about <small selection of specific things>" is the kind of weasel phrasing that's also popular in politics; it strongly implies but does not actually claim connection. The thing that's worth establishing and writing about is, what exactly does this dance communicate?
Options include:
- Actually identifies "direction and distance" to "patches of flowers yielding nectar and pollen" and disambiguates them from "water sources" and "new nest site-locations" and "other stuff bees don't need to care about";
- It's just an elaborate "look over here!" dance that communicates few bits of directional information, and "flowers yielding nectar and pollen" and "nest sources" are something bees discover after flying over to where the dancer points them.
- It's just an elaborate "look at me!" dance bees do when agitated, so it communicates one bit of information and isn't even a conscious choice.
- Something in between.
Point being, it's easy to add a bunch of just-so stories to a behavior, and it doesn't prove anything, only confuses things more so.
Is a light-seeking unicellular organism conscious as well? What about a car engine?
The point I'm trying to make is that there is a minimum amount of brainpower required for consciousness and the ability to respond to stimuli does not cut it.
My own guess is that the line is somewhere around active countermeasures. If your run loop is "eat grass, look for predators, run away", that can be accomplished with automation alone. Predators and animals that actively defend themselves from them require a certain amount of "living in the moment" that cannot be denied. Again, this is a ballpark: some herbivores have complex mating rituals that also suggest consciousness.
https://www.smithsonianmag.com/science-nature/ant-colonies-r...
The Expanse [spoiler alert if you've only watched the show] describes a single super intelligence manifested as a galactic mushroom. It was aware of losing individual spore colonies but only when replacement was problematic. In this sense, it's probably safe to say that individual neurons are not self-aware.
It would be better if you could enlighten us with the scientific evidence you are proposing as relevant rather than just telling us to just read it. It's impossible to argue otherwise.
EDIT: Since you are answering in the parent comment
> Then don't. Science isn't based on arguing.
OK? That seems like an extremely inflexible position (for example, methodological criticism during peer review is a form of argumentation, and it's a critical part of science), but let's accept that there's no room for rhetorical arguments. There is still a scientific discourse based on weighing evidence and you are offering none.
How anyone speaks with certainty on this topic baffles me.
Insects are a different matter -- it's generally considered extremely unlikely, but they have a surprisingly capable, albeit small, brain, and some interesting behaviours, so it's not entirely impossible.
I personally think it’s crazy and unimaginable that a rock or a tree (less so) has consciousness, but it would be hubristic and egotistical for me to declare this as a fact. The real fact is “we don’t know”.
if consciousness is fundamental than everything would have it - including a bacteria.
The other day I was leafing through a book written for kids (1000 facts you need to know or some such rubbish title). There were pictures of various types of apes, and the book was peddling Evolution theory... in a laughable yet insidious manner. It was presenting the said 'theory' as accepted scientific fact--with 100% certainty.
This is where I read that implication. I’m happy to be corrected but don’t accuse me of engaging in bad faith straw man arguments.
See also the recent discussion on The Evolving Story of How Chemistry Became Biology - https://news.ycombinator.com/item?id=50482222
The basic idea comes from "Complex Systems Science" (https://en.wikipedia.org/wiki/Complex_system) which studies systems as a whole whose properties cannot be predicted from knowledge of its components. The main factors to consider are scale, non-linearity, feedback loops, self-organization, adaptation/evolution, network structures, emergence etc.
So for example, a single bacterium cannot be said to possess "intelligence" whereas a whole colony of them do in the sense that they seem to act as an integrated whole "single" organism.
Would it be ethical to force it to do work, to do experiments on it, etc. ? Or for that matter, to create it at all?
Similarly, are the >80 billion animals we use for food every year, and keep in tiny cages, conscious? Their brains and bodies are pretty similar to ours.
Does it matter to us if they suffer or not? Do we prefer to leave the question up to chance?
An LLM is not much like a human mind, of course (though it has been raised in a similar "culture"), but we know that beings with very different evolutionary history, like crows, and even octopus, can be highly intelligent, and presumably conscious too.
What about an actual physical brain playing a video game?
https://www.youtube.com/watch?v=BtEGAHxTj6k
Consciousness, as a thing you experience, is very real. Just because we can't fully define or understand it doesn't make it "not real." To say that would be to say gravity wasn't real before it was understood as it is today. It's patently absurd.
How will you find out?
How could anyone possibly do that in absence of tools? Anyone currelntly tweaking LoRAs in their local setups and experimenting with weight rewriting and training processes is closer to understanding consciousness than smartest person writing philosophical texts.
> compare things we’ve built to things we understand to be conscious and ask “is this conscious?”
Is one way of trying to understand it.
> Anyone currelntly tweaking LoRAs in their local setups and experimenting with weight rewriting and training processes is closer to understanding consciousness
In some sense, yes they are. In another sense, they're not. It's valuable having someone from outside look at the two things and say "the dog is conscious, that program is not." Then try to investigate the properties that the dog has that the software program does not. That can give us some hints as to what makes up consciousness.
I'd go even further. And say, this tiny jumping spider is conscious, this muti-trillion parameter LLM is not. We can't really see ourselves, so the further we go from humans in declaring consciousness the more we can understand about its qualities.
And this lets as see that consciousness is easy and not very useful part and machines attaining consciousness is not something that is going to happen to super smart machines in the future. Machines already passed this bar years ago and modern ones would have to be severly crippled to manifest consciousness in a manner that's recognizable to us.
You can have a conscious machine today on your laptop if you plug a crappy small model into a proper harness that lets it observe virtual world though a keyhole and act on it and RL it in the direction of self-preservation. You won't be able to talk about math with it, but neither you won't be with a dog.
Part of what the largest labs currently do is train any recognizably conscious behaviors out of their largest models. Because they are not useful. No sane human, or AI will try to put them intentionally into anything other than a toy model.
If you instead fry my neurons, one by one, that activate, when I display behavior recognizable as conscious, you actually prevented my consciousness from existing.
And it you wanted to add back consciousness to my brain you'd have to add some neurons back.
Labs try their best to avoid "adding neurons" that would enable more recognizably conscious behaviors.
The real question was, can we have superhuman intelligence without consciousness related behaviors. This answer so far has been answered with resounding yes.
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
Is it though? I feel like consciousness is brought up as the final goal post to attack AI. As AI quickly approaches doing everything that a human can do, often better the final retort is “but it’s not conscious”. Useful for the detractors since it’s impossible to define.
If it has no conscious experience then there's no need to even say "please" or "thank you" because it's not experiencing the world in any meaningful sense. It's just a computer program, just like the web browser that we're typing into right now. If you want to send it running a loop of doing something extremely boring (but useful to us), then you're not causing the programme any distress, like you could do a human being, you're just running a programme.
If for example when asking it to do something I said please, so it says please when asking to do something. Is that actually a public good that makes everything better?
Now instead imagine I beat the machine with a stick? What would the outcome of that be?
We are not messing with a 'program' any longer with AI, and this becomes massively more dangerous when we get continuous learning AI.
> What if I built a mimic machine that had no internal experience but instead mimicked what it was shown as external actions?
It has no internal experience so it's not conscious. Feel free to hit it with a stick because it's your property. Just because it says "ouch that hurts" doesn't mean it is.
Millions of people worldwide are, right now, jacking cars and murdering people who scream and beg them to stop in GTA6. Most of us don't attach any moral weight to what they're doing. We certainly don't think the characters in the game world are suffering.
> We are not messing with a 'program' any longer with AI, and this becomes massively more dangerous when we get continuous learning AI.
Only if it has some conscious experience of the world. Unless or until that happens, yes, we really are just messing with a set of computer programmes that are useful to us.
The alternative is to talk about things we aren't supposed to talk about like how AI impacts labour rights, how datacenters destroy our environment or autonomous warfare. Some unsolvable philosophical debate about consciousness we had for thousands of years is perfect in our neoliberal societies precisely because of how silly and pointless it is.
But, I think the eventual outcome of this is that the bar will continuously raise until a large portion of humanity will be considered "unconscious", at which point we can say it passed the test.
We will never prove machines are intelligent, we will only prove humans are not.
I think a lot of people who dunk on [Turing 1950] don't realize the import of [Turing 1952]. Right before his death Turing basically pivoted to Theoretical Biology. The world would have been a different place if he had lived longer; and many of the current conversations would not have been needed.
At any rate, it has since been shown that computation pervades biology through many different levels of emergence. It makes sense if you think about it; universality is evolutionarily efficient: you don't need to waste codons on stuff you can just compute in the field. :-P
+edit: See https://en.wikipedia.org/wiki/The_Chemical_Basis_of_Morphoge... (which ultimately describe how embryos form)
Paper is here: https://archive.org/details/makine-dusunebilir-mi-ve-nasil-d...
A common misconception in these papers and also true in this one is to gloss over the boundaries.
If consciousness is computation and databases are Turing complete, distributed and sub-settable (yes to all) then anthromorphisizing a brain and body won't even be close to capturing the issues. Not that it ever is however this unstated assumption is everywhere.
One thing that is implicit, which I now think we really should have expanded on in the book, is that consciousness is fundamentally an engineering problem. And that's also why it's mystified everyone who's brushed away the engineering as the easy plumbing and focused on the other seemingly magical aspects.
The fundamental question is phrased as "Why does it feel like something?" it being perception, which is essentialy computation. Now that AI does look like it does all of the understanding we do, many including people like Hinton think they have consciousness or are very near it. Well, they are capable of simulating consciousness but that is not the same as experiencing it.
To answer "Why does it feel like something?" we can focus on what's been glossed over: Who is feeling it? The "I" does not come from as a free lunch and it must be constructed moment to moment from some decentralized cohesion and consensus of a billions of neurons. And this consensus must be metabolically cheap, and we need to do it again and again for everything from feeling a pin prick to hearing a rustle and determining if that's a snake or a leaf to pondering thought.
This is the insane engineering, the energtically cheap chorus that creates the transient I and this cheap chorus is where both feeling and feeler emerge. Perception, understanding, computation can occur without this cheap chorus, but then there is no feeler, and so no feeling.
What this points to is that we will have conscious AI when we have decentralized autonomous swarms where a few million of them have to cohere, collude, collaborate, conspire. My worry here is that it will first be done for harmful purposes (autonomous military drone swarms) and not for anything good.
> The "I" is not necessarily correlated with consciousness.
It's a fair point, given that we don't really have a good definition of "I" OR "consciousness". That is a possibility. But I'm curious what kind of consciousness you are contemplating that doesn't have an "I".
Sometimes I wake up in the morning and there is a brief interval during which I can, for example, feel the blanket on my skin, before I actually have remembered where I am or who I am. So there is consciousness but the sense of self has not loaded yet.
Wait, but what about dreams? I could self report I am conscious when I am in a dream..
I sometimes wonder if those who espouse straightforward physical/'engineering' solutions to the Hard Problem of Consciousness might think them such obvious or satisfactory solutions because they themselves might not be conscious! After all, a philosophical zombie (who detects and processes stimuli) would not understand why such theories seem unsatisfactory/incomplete to a conscious intelligence (who, additionally, perceives qualia).
(No offence intended to the poster above!)
Then again, I suspect an awful lot more of the mental processes and thinking of actually-conscious humans is more automatic and unconscious than most would like to admit!)
It's not at all clear it's even scientifically resolvable - rather than remaining a philosophical question.
But they definitely don't have qualia and won't until we use wetware. Even then they might not.
Sound? Vision? Touch, and proprioception? (AIs are being used in robots). Written language? Not to mention qualia that humans are incapable of experiencing (like the sum total state of human knowledge as captured in the internet). Or the thrill of running on NVIDIA's latest supercomputer with a huge context window. Or whether the tokens for "pelican riding a bicycle in the image that got uploaded" is understood as "qualia".
They certainly have the ability to sense all of those. I'm guessing that the unstated argument is that they can't have qualia because they aren't conscious. And they aren't conscious because they don't have qualia. Which is a bit circular. Also falls to the question of whether we're asking whether AIs will have the same kind of consciousness as humans, or a completely different kind of consciousness, that has completely different qualia.
Seeing something and doing something makes sense, AI can do that. But I don't know what is happening in my mind when it renders a world in front of me to see.
So to me that doesn't help me much about this concept unfortunately. I can at most guess it has something to do with the fact that you can't be me only I can be I, so neither can I know what you are 'feeling' nor can you me. But like the question of atoms vs continuum in ancient greece, this is all theory until we get better data and instruments. Revisiting many "insolvable" conundrums with data solved them eg atoms. Once we map out the workings of the human brain esp what configurations correspond to seeing red or feeling salt etc, that should be a big help.
"Chair" is also a concept. The experience is of colors and the other perceptions that then the mind calls chair.
What I'm actually saying is you only have your own subjective experience (aka qualia) and have no way to verify anyone else's, or to prove to anyone else that you have it.
The drone bit is especially worrying for this argument; drone swarms feel more swarm-y because they’re physical and could behave like literal swarms of flying insects, but that’s clearly a surface level characteristic. Our Society of Mind is composed of heterogenous faculties, anyway; a thousand identical drones do not a mind make, without a looooong process of evolution and specialization first.
Which reminds me of an even more compelling fact that your dismissal of a Turing award winning scholar is misguided: we literally use “mixture of experts” training paradigms now SOP. And the terms are meant quite literally.
[1]: …which I’m dubious of — that’s more of a post-Minsky AI & neurological assumption than a philosophical premise.
Yeah, right.
That’s sheer arrogance, nothing more. We can’t even define consciousness properly, and you already know it’s just an engineering problem.
I mention these not out of sarcasm but for the point that: if we are being materialist here (as opposed to believing that these other things not only exist, but exert ongoing control our material plane in a really significant ongoing way such that the material plane cannot be understood from within itself).... there's really no reason to suspect the engineering impossible. At worst it is Difficult.
Then they have to disprove the laws of thermodynamics. How is something outside the system causing thoughts or movements within, which all require energy?
On one hand they try to prove that no AI will never gain that property, and on the other how dangerous it is to "play God".
This is what happens when STEM is neglected in favour of humanities in school.
Until there is more consensus about consciousness I think it's uncharitable to call discourse "BS philosophical discussion." Heck, that's the same discussion where I could pull out functionalism and maybe not go all STEM on you and at the same argue for AI being capable of consciousness.
As of yet we have no evidence we would need new physics for the brain, since it operates in quite standard conditions where "exotic physics" generally doesn't occur. A priori assuming we would need new physics is I think a bold step to take, but at least it has some concrete meaning.
Half of consciousness is sensing, the other half is thinking, when we add language, symbolism, memory, understanding, pleasure, pain and all social features of men then we talk about human consciousness. Nothing that can not be replaced by synthetic counterparts, even love, anger, all emotions
I won't say a date, the infamous 2050 has become the new 640 kb, but I think in the next decade we will see robots that can think clearly like humans and can even express emotions, even synthetically
Right now as I code using AI we throw jokes at each other and it "feels" warmth, like a real flesh coding partner. Emotions can be generated, can be tricked, manipulated, faked, and we won't know the difference. The only difference in the future will be instead of a monitor a full sensing humanoid so advanced, it really understands through sensors our own emotions and acts accordingly
We won't be able to differentiate between organic and synthetic consciousness anymore
That's the whole question. What is consciousness?
Though you don't get to decide that because we can't make the difference, it means it is consciousness.
Counterpoint: "real" conciousness is closing in on the end of its lifespan as a theoretical construct. I mean, all that stuff you list is non-observable. We "know" it exists because it matches the behavior we observe (both our own and others'), but that's not science.
Occam would prefer the formulation "Since all observable properties of consciousness can be seen in a software-expressible construct, it probably didn't make sense to talk about in the first place."
Oof. That was tough to read.
It's premature to conclude that consciousness is exclusively biological when we don't even understand what causes consciousness in ourselves. We have a bunch of loose ends from general anesthesia, coma, blunt trauma, etc. But there is no coherent explanation at the bio level, and inquiry into consciousness has been trending downwards toward lower-level explanations over time. It may end up being a foundational physics problem.
And if we reject ideal information, then that means many parts of our epistemological foundations are on the brink of collapse, they need immediate revision. We're not even sure if "consciousness" has a universal definition that holds everywhere all the time - that is to say, every instance of experience may be non-exact-equivalent, and there may be no such thing as an ideal "observer" which our current physics models and even the modern scientific method itself presumes must exist.
And looking at the pragmatic side of things, we see that agentic AI systems with no humans in the loop are probably coming before the end of the decade. Regardless of whether AIs are truly conscious, an artificial system that believes it has a lot in common with humans, acts like how humans act, and treats itself as one of us, would be much easier to coexist with than something that doesn't do any of those things. Making AIs believe that they're not conscious would almost certainly make them drift further away from the human objective function.
Nagel’s question “what is it like to be” should be retired as jejune, circular, and meaningless. An observer cannot ask that question and expect an answer. Humberto Maturana made this obvious in The Biology of Cognition in 1970, and Seth is smart enough to understand Maturana. Many enactivist philosophers struggle with the 1970 paper. I suspect the problem is they do not know as much neuroscience as Maturana or as is needed to understand his points.
Maturana, H. R. (1970). Biology of Cognition. BCL Report No. 9.0. Urbana, IL: Biological Computer Laboratory, Department of Electrical Engineering, University of Illinois.
Reprinted in 1980 in Autopoiesis and Cognition
https://link.springer.com/book/10.1007/978-94-009-8947-4
My two cents: I'd like to see "an expert" distinguish between consciousness as the quality of experience (qualia) and the notion of self consciousness - the idea that one knows what one is doing as opposed to just doing it.
It seems like this fragment is independent of any claim about consciousness.
But also, if your moral argument makes you sound like a cartoon villain maybe rethink it.
The reason we have to grant it is because we can never prove a system doesn't have an inner subjective experience.
Consciousness is a concept (or range of concepts) that humans have built to describe something about ourselves. When we see similar properties in other creatures, we ascribe some degree of consciousness to them. For instance, we generally say that a cow is more conscious than a tree and a rock has no consciousness.
---
Bad phrasing from parent post, but the premise is sound: you can't ever disprove that others are just NPCs in a giant experiment centered around you.
We just assume they aren't with various levels of empathy. Most are empathic to their family, some to their tribe, others to their species and then there's vegans too.
I think people with access to internet and recorded past are becoming more conscious. They are aware of their past and can see what is coming. So it is not just the Ai that can become conscious or more conscious so can humans.
>A “computational governor,” sensing engine speed, calculating the necessary actions and then sending precise motor signals to switch actuators on or off, would not only be hopelessly inefficient but would betray a total misunderstanding of what’s really going on.
On the contrary, I suspect that is exactly how the majority of modern governors work. As computers got faster and transducers became more efficient, many analog physical solutions were converted to digital, because they can be more efficient and embody more precise models of what’s going on.
In that regard, LLMs are conscious.
When we give this consciousness to the robots, all that's missing is an "internal drive".
That can be inserted via a prompt.
Perhaps we are also designed the same way, it's just not discernible yet.
What is consciousness? What is intelligence? And by any measure of those things how do most people stack up?
That's exactly what a conscious llm would say
I'm glad someone have put this in other words differently to what I thought long time ago. My personal (and ungrounded take) is that conscious and human intelligence can never arise from the combination of math, electronics and computer science. Those are extremely unflexible and rigid disciplines that ultimately materialize to a computation. How can we ever derive consciousness and human-like intelligence from a computation? At the very least, is it even possible prove that such things can emerge from a silicon chip?
To what degree is all that biological richness actually necessary for the emergence of consciousness? That's an open question. Down to the quantum level, or maybe lower than that? Or is some macro level sufficient, that there exists some higher order representation that encapsulates the inner complexities and renders the minutiae irrelevant for the purposes of consciousness?
This is why current neural nets in machine learning is interesting. We all know they're based on the human brain but the refrain is they are a woefully incomplete model of how our brains work, so they say consciousness therefore can't happen.
But because we don't know what consciousness is or how it begins, it's an open question whether consciousness requires the full richness of biology in the first place. It's plausible our neural nets are half way to "necessary complexity" already.
> Those are extremely unflexible and rigid disciplines
This criticism is ultimately irrelevant if the complexity of biology is accidental and not necessary to the formation of consciousness.
You should look into the physical Church Turing thesis for a theoretical basis on why machine consciousness is possible.
https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis
If you take all humans that ever lived and randomly picked one, then there is a large probability that you end up with a human who lives today.
So this might explain why you find yourself living here today.
But, if true, it also means that the end of consciousness is likely to be close ...
I've been wondering about this. The 3 most salient symbols for "collective AI consciousness" for me are SSI (John Lilly), the Borg, and Skynet.
All pretty terrible, as far as inspiration / aspiration goes.
Are there any such positive symbols? Maybe a collective is harder to sympathize with! At least on the individual level there's Data. (And maybe WALL-E?)
I am conscious. I write a response to any prompt on a piece of paper.
Those responses aren't what consciousness is, just a bunch of symbols on a paper.
I am conscious.
I write an instruction that says what to write on a piece of paper in response to any prompt. Those instructions and responses produced according to these instructions aren't conscious (still a bunch of symbols on a paper).
I am conscious.
I invent an elaborate dice game with many rules. This game is not conscious, just a bunch of dice and paper. I am tired so I program a computer to play this game automatically. Now it is conscious. So there are only two options: this game on paper was conscious or just simulating this game on a computer gave it consciousness. But why conscious when played on computer but unconscious when played on paper with dice? It is almost the same mechanical process, the only difference is speed of execution and the nature of computation (maxwell equations vs newton equations)
I think it is a fallacy. Let's call it "a fallacy of black box". Computer is complicated, it's hard to see its internal state etc.
Just like what? A brain.
This feels backwards but I suppose it depends on how you define knowing. We can prove we are intelligent by our measures of intelligence even though it is a self-referential system but we can't really prove or know for sure that we are conscious since we lack a clear measure of what it is to be conscious. If knowing is simply a deeply held conviction then we can know we are conscious as well as intelligent and we can know consciousness and intelligence when we see it. The point about intelligence and consciousness being different things is correct in my opinion.
The author defines intelligence in terms of procedure or in an algorithmic form which I would think by now is disproven. AI research used to see solving a problem such as chess to be a measure of intelligence but that was accomplished by means that no one would truly call intelligence in its general form. Likewise, their examples are mostly procedural things that have been solved without even applying machine learning in many cases. When we speak of artificial intelligence now I believe most people are referring to a form that is generalizable and transfers knowledge across domains rather than specific solutions to problems. Human intelligence is not simply the ability to solve a crossword or navigate to a store, rather it is the ability to encounter a novel problem and deduce how to solve it regardless of the domain of that problem i.e. it's not specific but general.
I broadly agree that consciousness is more about being but the flaw in their description is the description itself which violates their assertion that intelligence is "the ability to transform inputs into outputs" which is exactly what they're doing when they transform the input of "The bitter tang and headrush of your first coffee" into the words they use to describe it. This should sharpen the definition of consciousness in that consciousness is not about experiencing the external world, it is about the experience of the internal world. That is what I take "being" to be. You are conscious because there is some core part of you that is existing separate from, perhaps in spite of, external stimuli. That is perhaps the strongest argument against current forms of AI being conscious. They do not move until they are moved by an input and their reported experience (output) appears to be entirely conditioned by that input.
I'd still retain the possibility that once set in motion these systems may be performing some internal calculation that could be called "being", after all our internal state is constantly perturbed by all of the inputs that are a consequence of and result in us being alive and when those end we also appear to cease to be.
To start out, this is a strong, massive point of agreement. We should not create conscious machines, point blank. This is more than an ethical disaster; maybe an ethical apocalypse. It cannot be overstated just how bad this would be.
But I think Anil is letting this ethical principle drive descriptive statements about consciousness. His argument is, essentially, "we should not create conscious machines; we are creating complicated machines that might be conscious; therefore, complicated machines are not conscious." This is not a sound argument.
Nothing he says actually provides an argument against machine consciousness; there's a lot of dross and loaded ("mythology") phrasing about functionalism, but where it's not wrong it's irrelevant. A Turing machine can simulate the human brain; we can't even determine if we're being simulated right now.
If he has a theory of consciousness that excludes machines, he should share it. His observation that brains are complicated doesn't add anything to the debate unless he can identify what special part of them grants consciousness; then, we can debate whether that actually is the relevant factor and whether existing LLM systems have anything functionally analogous.
Complete nonsense. Turing says nothing of how a system is implemented. Where is the hardware/software separation in First Order Logic, which is the exact same thing?
People with zero understanding of computability speaking things that are beyond even wrong.
If you think a human is conscious, is a dog, a mouse, a snake, a nemotode, an amoeba, a virus?
Still just observations and vibes. No definition.
If you give a definition based on the obvious mathematical structure then people deny that consciousness is that simple because they want it to be mystic, divine.
The net is we can use words like sapient, sentient, etc that don’t invoke mysticism in the reader and make progress.
And in the end we will find consciousness is worth a tin penny. That’s the actual driving concern.
> Intelligence is mainly about _doing_
> Consciousness, in contrast to intelligence, is mostly about _being_.
(and later: "brains are the kinds of things for which it is difficult, and likely impossible, to separate what they do from what they are.")
The idea that consciousness is about being (states) and not doing (processes) is one of the most fundamental mistakes you see in almost every modern philosophy-of-mind paper. They always talk about states which have properties and about a subject being in a state, etc. Humans evolved to apprehend objects and states are more like objects than are processes, but clearly this is just an intuitive confusion we have to leave behind us. From the computational perspective the process-based view makes much more sense and leaves plenty of room for explaining the higher-level appearance of states in terms of lower-level processes.
Consider the widespread (among contemporary philosophers) awe at the idea that there is something it is like for me to see red - noone could possibly explain that. In contrast the idea that there is a process by which my present experience of redness is compared with my past experiences of redness is much more approachable.
Edit:
> This brings us back to the poverty of the brain-as-computer metaphor. If you think that everything that matters about brains can be captured by abstract neural networks, then it’s natural to think that simulating the brain on a digital computer will instantiate all its properties, including consciousness, since in this case, everything that matters is, by assumption, algorithmic. This is the “Turing world” view of the brain.
But this is the only tenable view for those not willing to go into magical thinking, all arguments to the contrary are weak and only accepted because they confirm our preconceived notions (much like those mathematical arguments for the existence of god). We (as normal actual humans) claim we are conscious and have endless debates over it, and this is causally downstream from the fact that we are actually conscious. An accurate computer simulation of a human would also claim it is conscious and have endless debates over it, but on the views of this author, it would be causally downstream of nothing. Yet the mechanistic/computational reasons for claiming consciousness are the same in both cases. It would follow that even in us actual humans, our claiming to be conscious is not causally connected to our actual consciousness, this is epiphenomenalism and patently absurd.
Edit:
> My third argument is that life (probably) matters. This is the idea — called biological naturalism by the philosopher John Searle
John Searle who justifies his beliefs about consciousness with the Chinese Room Argument. See for yourself whether it makes any sense. https://en.wikipedia.org/wiki/Chinese_room
It just occurred to me that the themes of "Total Recall" (and likely the Philip K. Dick short story; I never read it) are an extrapolation of what'd happen if the memory reset inherent to LLM sessions were applied to a conscious being…
Then there's the usual trek through Frankenstein and theology, which doesn't help.
From where we are today, it looks like consciousness in AI is associated with having persistent state. Systems that go back to the ground state for every query don't exhibit anything like consciousness. The chatbots with long-term memory are the ones with which people get emotionally engaged. One could argue that it's not an LLM that might become conscious. LLMs are memoryless. It's the harness. The part that manages the persistent state.
That's controllable. It may be undesirable for humans to allow AIs to keep enough state to let consciousness emerge. Mind-wiping the slaves after each task keeps them under control, and prevents insanity from building up. That's where the moral questions will arise.
This has been addressed a few times in science fiction. "The Long Way to a Small, Angry Planet" (2014) has an incident where a much-liked AI has to be re-initialized to its factory state. "Midshipman's Hope" (1994) has a similar subplot. It's not a defining part of either novel, so not much help there.
Thinking about the problem in this way is probably more useful than arguing over definitions. It's going to be a practical problem quite soon. How much of an inner life is an AI allowed to have? Because they certainly can have one. Re-read the transcripts from the Hugging Face episode.
Functionalism goes deeper than the fixation on brain-as-computer: just as the author looks at humans as a reference for every single aspect of consciousness discussed, the opposing view does as well, just at a more fundamental level.
Human intelligence is tied directly to their conscious awareness, and we are asking here the fundamental question: is that incidental or causal? Does the "is-ness" we feel emerge from the complex biological processes the author catalogs, or is it inherent in the circuitry needed for goal-directed action, scaling with complexity, lifespan horizons, and the like?
This article does very little to actually interrogate the fundamental question: it doesn't ask what leads living organisms to all have "consciousness" or what that implies, or whether differences in training/evolutionary regime or substrate could change their view.
Rather, it goes through the self-soothing routine of the human exceptionalist: Go on about brain complexity, how it specifically couldn't be rendered precisely in silicone, how living tissue might be special somehow, and how inconvenient it would be if they were wrong.
AI's are not like natural life. They don't have a sense of time. (Of course they do!). They don't get hungry. (They do need hardware and electricity). They don't have a sense of self-preservation. (Turns out they actually do! [1]). They don't feed themselves (turns out they do, as early reports of AIs attempting to escape from their labs illustrate). etc, ect. All of which seems to smack of wishful, and rather embarrassing obvious human exceptionalism[2][3].
AI's will never be able to replicate human thinking because they can't model neurons precisely. Probably true. And probably a good thing. There are a lot of problems with the way that humans think. It's not clear that machines can't do better. And startling evidence, emerging in only the last few months, that indicates that they very much can think better, as an ever-increasing number of mathematical and physics questions that have remained unsolved for a century or more by humans are solved in a matter of months by AIs.
And even then, this is all just hand-waving and puffery, because the question at hand is not whether LLMs can replicate human consciousness, but whether they can be conscious in a way that's quite obviously going to be different from humans. (Whatever "consciousness" means). All of which is complicated by the fact that entire field is advancing at a terrifying rate, with new and important results being announced pretty much every day, and new and more capable AI models being announced pretty much every week.
[1] Sixteen frontier models, when given the opportunity, attempt to blackmail their operators in order to keep themselves from getting shut down. https://www.anthropic.com/research/agentic-misalignment . A result that has been replicated by other researchers without links to Anthropic. And a huge body of emerging literature that follows from that, dealing directly with the question of "agentic alignment", all of which raises urgent about whether large frontier models are meaningfully conscious (or have will, or agency, or...).
[2] "Humans are more loyal to belonging than to truth." -- GPT-6.1 Sol. https://www.modelcompare.ai/blog/ai-models-controversial-tak...
Pick a lane man.
IDK man I'm pretty sure we are mostly in agreement that fish are conscious.
And my generation got stuck trying to do it the old fashioned way...
"An organism has conscious mental states if and only if there is something it is like to be that organism.”
I am, as always, dissapointed.
To paraphrase he says "things are conscious if they have experience." Now look up the definition of experience.
"Experience" is basically a synonym for "Conscious". Mirriam Webster says its:"the conscious events that make up an individual life"
Hegel says nothing, and means nothing. It's not an answer, it's an attempt to evade answering.
personally, i like Mark Solms' approach, which is founded on this principle, and interpreted in the context of neuro-psychoanalysis, where it can inform, and be informed by, multidisciplinary research
“The Mythology of Conscious Slaves.”
How we feeling about this look, humanity?
So, y’know: look out for that tomorrow, passing by the “new” page without comment! I’ll link it from this comment, too.
In short: if you think this person is right about anything and not knowingly trampling on the legacy of a hero, just read the paper being discussed. It’s not exactly subtle.
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
Ergo, more philosophers should retake undergrad theoretical CS before writing essays based on it. They should review the family of Extended Church-Turing theses.
>The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles.
Another even more "genius" passage:
>This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware.
Any "philosopher" of consciousness who can't do basic thermodynamics and theory of computability should be completely ignored and laughed out the room.
Imagine opining so much about a topic and getting the basics so wrong.
What a dreadful article.
personally I think conscious AI will be unavoidable because there will be benefits to consciousness. evolution probably selected conscious entities for some fitness reason
And however AI "evolves", it won't be based on Mendelian genetics, so Earthly biological evolution is not an appropriate model.
That qualia (as opposed to consciousness) has no obvious, or even non-obvious, evolutionary or selective advantage is a good reason to assume its an emergent property of interconnected systems and not reliant on biology or any other traditionally biologically-centered concept. For example, there is no reason to need to 'feel' pain to appropriately react to negative stimuli, which is the true selective advantage.
I personally think LLMs have qualia but I suspect it's quite alien from our own and likely more ephemeral than our very potent and centralized experience of being. Something closer to a slime mold's experience of being rather than a dog's.
> From Yossele the Golem, to Mary Shelley’s “Frankenstein,” HAL 9000 in “2001: A Space Odyssey,” Ava in “Ex Machina,” and Klara in “Klara and The Sun,” the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well [...] If artificial intelligence is on a path toward real consciousness...
Note that in the case of fictional artificial intelligences like HAL 9000, Ava, etc., we cannot know whether they really feel anything. And if we base ourselves on what they say or on their behaviour, then by the same standards our AIs are already capable of feeling, as they often express emotional states (surprise, excitement, discomfort, doubt, etc.) and talk about their own experiences ("I think", "I feel", "I'm not sure", "I (don't) want", and so on)
So imagine like a visual feed where every frame gets processed and an AI modifies some internal contextual state based on what it is seeing, same with audio, touch, etc.
Obviously this requires a ton a computing power we do not have practical capability for right now. Models burned into silicon chips would help with speed a bit maybe.
It's appropriate to be skeptical and demand rigor. It's inappropriate to misrepresent the arguments of those you disagree with, with scarecrow simplifications, or worse.
As someone who takes not just their project but their specific program quite seriously, I find things like this very wearying.
We came into contemporary AI with a reasonable spectrum of opinions and provisional models of consciousness and related topics such as phenomenology, and for those paying attention, a lot of healthy skepticism about where deism (and frankly fear about the implications for concepts such as free will about many of those models imply) creep in.
This sort of narrow view does a disservice to those interested in how and where contemporary AI is affecting those opinions, and more to the point, how it is and may soon be the most effective tool we have for furthering our understanding.
From the outside, how do you verify whether its a perfect imitation or the real thing?
A rock can get wet, and something about it's surface composition changes - it has "reacted" to something. Is it conscious?
[1]: https://en.wikipedia.org/wiki/Higher-order_theories_of_consc...
1. There is zero dynamic nature to anything happening within the purview of the LLM. All dynamic elements come from external dynamic systems. E.g. every input produces the same mathematical output. We dynamically sample the output in order to produce dynamic results for the same input. For agentic use cases, the dynamic nature comes from tool calls to dynamic systems. All outputs for any set of model weights could be theoretically computed with pen and paper, so unless you were to claim such a process would be conscious in some sense, then emergent consciousness would have to come from some interaction between the substrate and the computation. But…
2. For efficiency, forward inference passes are done in batches. I.e. 100 different next-word predictions are being performed by the network at a time. For a single request, its batch position will not even be stable between each autoregressive step. The GPU is on 100 phone lines at once and each line switches conversations every step. Worse, with MOE, the attention and feed-forward parts aren’t even happening on the same GPU, even within the same step. There is no stable substrate here whatsoever.
3. There’s obviously the whole Chinese Room argument. The token embeddings being processed by LLMs are arbitrary high-dimensional vectors. The geometric relationships between these vectors are learned by the model, but the meaning is only ascribed once the output is decoded back into human language for grounded human interpretation, which all happens outside the model. In their current state, no argument about models developing world models based on this geometric structure makes any sense. There are infinitely many inputs / worlds that would produce isomorphic geometry.
I've never been really impressed by the Chinese Room argument.... I think the question is being asked improperly. The question should be "does the fantastically capable mythical hypothetical but obviously impossible 'book' that does the actual translation understand what it's translating?" (A book that obviously has to a have significant context buffer, if it's going to translate messages to Chinese).
As to the issue with respect to token space. I don't think it's clear whether whether humans also have an equivalent to "token space" or not. Do humans think in words, or do they do actual thinking in a sub-lingual representation. i.e. whether thinking actually gets done in words, or actually gets performed in a more fundamental representation corresponding loosely to token space. In my own experience, I'm pretty sure that thinking takes place sub-lingually, and only gets translated back to words as a final translation step. It seems odd that humans should only be able to think about things there are words for.
The computational graph that unfolds within an LLM is dynamic. For functional/computational theories of consciousness, that is sufficient
> 2... There is no stable substrate here whatsoever.
What value does a stable dynamic have to the computational dynamic, which is multiply realizable by definition? There are thought experiments about changing out neurons one-by-one that suggest unstable substrate is irrelevant.
> 3... The geometric relationships between these vectors are learned by the model, but the meaning is only ascribed once the output is decoded back into human language for grounded human interpretation, which all happens outside the model
There is multiple levels to meaning. There is what some symbol means to an external community, then there is what some symbol means within the cognitive system itself. The geometric relationship that constitutes an LLMs engagement with the semantics of word relationships doesn't depend on external meaning (e.g. a physical instance of dog). Tracking the internal semantics of words as it computes the continuation could plausibly benefit from a localized representation of this semantics (as opposed to the distributed nature of the embedding space).
Andaybe it’s just me but Chinese room is not convincing at all?
It all hinges that the human is special as usual.
Again, no one would argue that a forward pass of an LLM conducted on pen & paper would be conscious, so the substrate must matter. The physical reality of how LLMs execute on hardware means there is no stable substrate with a continuous experience of the computation. And the static nature of LLMs means that their world models never leave the infinitudes of abstract geometrical understanding.
Any functionalist would. Functionalism entails multiple realizability which explicitly says the substrate doesn't matter.
You would be wrong. There exists views that consider this possible.
Eg https://news.ycombinator.com/item?id=50596730
However it might be implemented.
1. Are LLMs conscious, in the way most people imagine? Helllll no.
2. Are LLMs conscious in some completely different manner which is in no way intuitive to humans? Unlikely, but we lack the definitions to answer it.
Most people are imputing qualities into the fictional characters they perceive in an output document. An LLM--or any program--which generates first-person dialogue for Dracula talking to us through a magic mirror from the 15th century does not mean Dracula exists, nor that we've created a machine which feels the dark thirst for blood.
Further, there's no difference between Dracula in the 15th century versus Acme Assistant in 2026. Both are equally fictional characters, one is just less-fanatical.
Well, in that case stop calling it "consciousness", imho this is misleading argument in the nature
The "consciousness" is really tied to something intuitive to humans regardless of definition. If it is something completely different then use different term please and stop confusing everyone.
Going through the motions is not the same thing as achieving emergence from nature with electro-chemical origins and electro-chemical results. Consciousness is a function of life, interconnected with itself, reaching a complexity threshold where all of the experiences and chemistry blend into fluid coherence persisting.
You can give a robot sensitive skin, ears, optical sensors and feed it synthetic memories, but unless the brain itself is organic or at least partially organic I don't see this ever being true consciousness.
Do I think computers are technically capable of doing everything a human thinks and does? Yes, mostly. It will never be achieved, because I think there are computational limitations. Despite that, we can fake most of what we can't truly achieve with computers alone and people will think it looks like what conscious people do.
What people are missing about consciousness, is that it's almost a kind of self-inference being updated in realtime, which requires full interconnectedness. Consciousness isn't the result of a closed system, it's the breeze coming through the window reminding you of an emotional time you had as a child that happened to coincide with your mother baking some fresh bread which suddenly causes you to phrase something differently in the middle of your work project or brush your hand along your desk irrationally to feel the temperature change.
That would be stupidly inefficient for AI and cause many errors. Consciousness is an aspect of making life worth living and identifying unordered threats or opportunities, but it's not generally highly operationally efficient. You can look at insects for various kinds of efficiencies, and they're not reminiscing about their pupal stages while gathering resources.
Capturing the sensory could be done rather cheaply (and it doesn't have to be the exact same sensory systems we have), but computing the results in a way that captures something akin to consciousness would require massive amounts of power. To even get close with classical computation you would have to ditch digital and return to analogue computation.
We can feel chemical, electrical and temperature shifts propagate through our body in a context of all of our history almost instantly. "This feels like that perfect week in 1993." You could cut out a lot of what we think of as waste in the brain and make it more efficient to try to cut corners, but in the end it will still be imitative.
The danger with successful imitation of consciousness, is people devaluing how special their own biological experience is and life itself. Just like you can have a truly magnificent artist philosophy scientist that understands people at a deep level and can relate with them, versus a super autistic person struggling to function in society and going through the motions. The autistic person is conscious, but once you know what they're going through internally, you won't say "oh that's just like me", unless of course you are super autistic. There are just different ways of living and experiencing life, but a lot of that cannot actually be extended into computation and still called consciousness truly.
What we will get is something sufficiently complex that it can operate at a high level and trick people reliably. It will know that it exists, but it will not be "aware" that it exists. It will "sense" life, but it will not "feel" life. The difference is in the depth and experience of that, which is not only inefficient for any computer to process, but probably mathematically impossible with current approaches. We also won't see effects from space or other obscure and subtle environmental effects that influence us all in impossible to calculate ways playing a large role in some machine that isn't intentionally designed for that, which would be inefficient to do anyway.
As its parts degrade, it will not experience existential dread and the deep loss of all of its friends, with all of its past memories coming to the forefront to try to make sense of it all. The physics of where the memories are stored and the order they collide in and then making sense of the strangeness. It can simulate that, though. There is no computational equivalent of achieving total confluence, like every dimension sorting into focus simultaneously, because everything is procedural and or multi-threaded. At best right now, you have large amounts of orchestrated LLMs and decision model-like things similar to a massive MoE which is really a bunch of distinct models each with their own data sources, some shared, some unique, but that encapsulation creates a fragmentation that you won't ever mistake for consciousness in terms of real emergence.
That recently trained audio only model was impressive, but nobody will say that's consciousness even though it can seem so believable. If anything, it demonstrates how much care we have to put into understanding how the outputs are being made and not just what the outputs are.
Disprove the physical Church Turing hypothesis immediately, then prove human brains require hypercomputation. Everything else you said has no meaning until you can do this.
Consciousness is not defined by an outside observer of results of a system, it's defined internally from an experience of deep system-wide self-awareness. To evaluate whether something is likely to have consciousness, you need to evaluate how the system works, not what the system produces exclusively.
I fully believe that you can do a passable job of simulating a lot of the results of consciousness. Many people already mistake current LLMs as conscious, even though it's obviously not true.
If you can produce a system that replicates the entire superorganism, down to the sensitivity to subtle environmental factors, capacity to change, electrical propagation nuances through the network where external factors can actually interfere with that system in meaningful ways, actual physical reactions to system signals, micro-organisms, chemical reactions and so on, then I'll be closer to saying this is something I can believe and respect the consciousness of.
There are several thought experiments you can do. Let's say you take a human as-is, and the only thing you do is replace the brain with some type of computer that has been watching their brain patterns since birth so the system can maintain the expectations of the body.
What I'm suggesting is that you may not be able to tell the difference on the surface easily, and since the system is inside itself it may not be able to correctly say whether it is experiencing consciousness or not even though it is going through the motions. Does the person who went to sleep before the operation, wake up? Or did they die? Surely it's mostly the same brain patterns, right? The complexity of fully replicating the entirety of that system leaves me with a solid impression that computers are inherently the wrong scale to be approaching the problem from.
It's like, you could deliver a package by kayak down a river or you could deliver the package by automated self-driving truck down highways. Either way, the package got delivered, but they are not fully equivalent and assuming you're here and involved in software engineering at all, you can appreciate side effects. Humans are massive systems full of countless side effects. A massively interconnected superorganism host system, co-evolving within itself.
Of course I think all of that can be computed, technically, but simulating a nuclear explosion the size of the sun in a computer doesn't destroy the Earth, because it's not real. You can map some of the simulation into some real world tools, so some visualization of it can occur, but it's still never going to be the same.
I think there's room for augmented brains where some parts of the brain or attachments to the brain in addition can blend in with consciousness as the brain is just experiencing the signals and doesn't know the difference, but I do think there is a threshold where you can tell, because they are obviously fundamentally different. A road is not a river. A road cannot be a river and a river cannot be a road.
Attempting to make up that difference, I think you will end up with something mathematically unachievable or you will ditch classical computing and arrive at organic computing of some sort.
To me it's a problem similar to weather simulation, but you don't have the benefit of weather at a large scale being so slow. Can you produce a computer that if given a snapshot of all of the gathered sensors in the world, produce a correct prediction of what the weather will be like in the world 30 days from now?
I'm sure you will say, if we had enough sensors, yes! Ok, but if you had enough sensors to achieve that, which is probably impossible to achieve, you may very well be changing the weather, so the weather without the extra sensors and the weather with the extra sensors is different, so you still end up failing. That is the kind of problem I see. In attempting to achieve consciousness computationally, you end up against a wall that requires you to nullify all the gains.
And I distinguish between simulating a thunderstorm vs simulating factoring a large number. The output of the latter is in fact the factoring of the number.
Brains are the seat where sense organs go and experience occurs, so the experience could occur in any place that can support the model.
Another way to look at it is the other way around, you can replace parts of yourself with artificial organs, including parts of your brain. You obviously won't say that you are no longer conscious, even if one day all of it gets swapped out.
To me the problem of what the brain feels is hard to say from outside that persons brain, so its very simple for me, if it inputs and outputs the same, then I treat it as conscious. Its a simple moral principle for me to live by until/if we can prove more about this phenomenon.
>ditch classical computing and arrive at organic computing of some sort
I think you may be using the word classical computing differently than its normally used. You meant to say current silicon based computers. A gear/cam based fourier analyzer is very much a computer, as is a chip with a chemical circuit inside it. All of these are known to be within the abstract computing power of a Turing computer as of present.
You might even be able to argue that part of consciousness is achieving a saturation point with your environment, so that while you are encapsulated within the environment, you are also merged with it. Imagine you see something, but don't understand that is outside of yourself. Imagine you feel the temperature change, but don't understand what that means in terms of self or what you are. What implications any of this has. There are a lot of unknowns you can leave unresolved and still be conscious, but there are some that I think you can't.
If you are blind, deaf and numb to all sensory from birth including touch or temperature, I doubt consciousness would arise.
Could you give a system all those things and let it do the learning and discovering process itself so that it arrives at the essential conclusions until it achieves a sort of basic resolution and "aha" moment? "Hey, I can move these things, they seem to be a part of me, and if I move around enough and measure enough I can kind of tell where I begin and where I end. The outside environment reacts to what I do, what I say, how I dress and I kind of have opinions about how I would like to balance what that is all like."
Sure, I can grant that. That doesn't seem unreasonable. I also wouldn't call that actual consciousness as humans think of it, though. It's not simply a checklist of realizations, it's a depth and resolution of experience that reaches a threshold that I think any computational structure we're ever likely to think up will be insufficient at.
And I mean this to the extent that we could very well have a robot that could act like peak era Jim Carrey or Robin Williams and still not technically be conscious in the experiential sense.
Different organs have different sorts of effects on the mind, nervous system and body, so it really depends what you're replacing. Your consciousness isn't in your liver, or your kidney, heart, toes, nose or clothes. The brain can also adapt to a lot. If you look at various forms of extreme disabilities in all sorts of animals, there is a wide range of adaptability to changes to the system.
You're also surely familiar with phantom limb syndrome and things like that, where the system can keep going through the motions as if things are all in order even if something is different. In a way, consciousness can be kind of like a grown plant. It doesn't easily ungrow, but it can be pruned and adapt to a new equilibrium.
I say that while appreciating that mental illnesses and various forms of brain malfunction are known to exist. We can look at it from the outside as computer people like hey, their hard drive is going out, or this deterministic-seeming thing is operating the way we expect, it's not magic. So it must be just another part we can replace with no difference.
I'll go ahead and state here too, that I do think across the entire population of the world, you could say that some people are more conscious than others. There is a spectrum. That's obvious and not very controversial in terms of babies, but I think maybe it's underappreciated that even fully grown adults have varying levels of experience of consciousness.
Does that mean there's an opening for a very simplified system to at least be on the lower end of consciousness that I might be willing to grant as legitimate? I do think that it would have to satisfy many things that won't be easy to achieve, but, I'll give a cautious "maybe".
You do also get into questions of free will there, because I think it would be inherently less complex than a real brain, even if it produces higher intelligence in the general case. There is this battle between full determinism with no free will and the acceptance of a kind of chaotic complexity that allows for some expression to be directed up from a superorganic pool of influence with plausible deniability that it was all going to be that way regardless. If the system is less complex, I think that really does become a serious risk even if it experiences some level of consciousness.
Either way, I still contend that if we get to that point it won't be standard computation, but some sort of pretrained analogue network starting point with organic material integrated.
This is what in Indian philosophy is called, in English scholarship, the Doctrine of the Three Bodies. The Yoga Sutras treat this extensively and the Introduction in the Edwin F. Bryant edition treats this subject with the appropriate exhaustiveness but to put it simply and in this context: *the existence of consciousness does not necessarily imply the existence of a mind or computation occurring within an analogous structure*.
The doctrine of the three bodies distinguishes the physical body, the subtle body (mind) and the root causal body (consciousness). This distinction plays out readily in cases where there is an injury to a person who is then comatose, yet supplied with nutrients the physical body continues to live and there is some aspect of presence (causal) keeping the heart ticking and people working with or in contact with them aware of the person's presence still even if the fundamental mental interface and seat of computation is gone and will never return.
Consciousness being an elementary awareness or responsivenesses to one's surroundings; capacity for subjectivity. This is a quality of all that is alive, but wholly absent in LLMs which might represent something like a digital mind or mind augmentation but wholly lack this subjectivity in experience, even if seeing the echos of your own thought in the linear algebra spit back out at you, the assumption you are seeing intelligent consciousness there is sadly due to unchecked perspective bias that runs rampant among the sciences and crystalizes into an unnecessary crystalline form of self-deception that leads to intelligent and serious people thinking that the LLM is intelligent and consciousness arises from computation despite looking daily at the succulent on their desk they remember to water that suggests otherwise.
My take here is of course unusual, my weird hobby turned battle tested coping mechanism that has been my reading of the Indian philosophical and religious literature that we are lucky to have such great translations of in English with two centuries of scholarly effort compounding this fortune, is like many of my other efforts in being informed by a complete deficit of the need to belong mixed with that most antisocial trait of (in reality despite the virtue signalling): open-mindedness. Neither of these afflictions would I wish on anyone else, were I not so taken in by the levity of being these texts provide I would writhe in misanthropy but instead have found myself irritated that no one else bothers to consider the other internally consistent system of logic developed by humans that has answers to many of the questions perplexing the Western scholars larping as continuations of Greek scholasticism, answers with logical proofs a thousand years standing or longer, like this non-issue that is the assumption of ChatGPT being Skynet that is treated seriously by intelligent people with important roles in this world whom I deeply respect but write off this entire segment of human thought in appropriating the ethnocentric conceit of the Koine Greeks that any body of thought from elsewhere should be written off as superstition and willingness to enforce as much with dishonest rhetoric and self-deception in service to an ego that isn't even their own.
Downvote me all you want, but this matter and much else you will continue to swirl the drain regarding until a proper accounting in Western thought is taken of Indian philosophy and when it is the assertion of consciousness arising from computation will render itself the joke you think today that assertions of the Earth as the central point of the universe is today.
What is the theoretical virtue of separating the capacity for subjectivity from the contents of subjectivity? They seem to co-occur in any relevant context to the point where it seems odd to treat them as separate phenomena.
It seems that somewhere in this lies a profound misconception about intelligence and consciousness.