Google CTO: A unifying theory of life, intelligence, and technology

This content is locked. Please login or become a member.

From von Neumann's blueprint, which predated the discovery of DNA, to viral genes that now help form our memories and placentas, Blaise Agüera y Arcas argues that function defines life more than material does, and that AI belongs in the same story. He explains why he thinks AGI is already here, and gives a blueprint for using AI without sacrificing your own mind.

My name is Blaise Agüera y Arcas. I am the CTO of Technology and Society at Google and the founder of an organization there called Paradigms of Intelligence, which does basic research on AI. I also have just published a book called What is Intelligence? Today on Big Think, we're going to talk about life, functionalism, and how to live with AGI.

So after thinking about it for a while, the definition of life that I've come up with, and I should caveat this, that every biologist's definition of life is going to be different. This is not a term that there's any consensus about. We actually wrote a paper about that in which we interviewed all kinds of interdisciplinary researchers. And for 60 experts, there are 60 definitions in there. But mine, for what it's worth, is that life is self-constructing computation. That complexifies through symbiosis. So what does that mean? Let's start off with self-constructing computation.

A lot of people, when they hear computing, they have associations with a very specific technology that began in 1945. They think of it as an applied science. And computing, as Turing and von Neumann thought about it, is actually something way more fundamental. It's a basic science in the same way that thermodynamics in the 19th century, which describes phases of matter and so on, is a basic science and long predated steam engines.

So Alan Turing developed a theory of what computation is. He did something really interesting, which was to look at computation, which at the time was done entirely by humans, and define a mathematical abstraction for what it means to do this kind of computation. He stripped down the system. system to something very, very abstract, to just a read-write head that can read and write symbols on a tape called a Turing machine.

Now, the real magic in Turing's mathematical proof is that he then said, well, You know, any computation can be defined, you know, with the right table of rules. It could be doing whatever. It could be adding numbers together. It could be taking square roots. It could be, you know, figuring out the statistics of a whole corpus of text, you know, doing the kind of things that an LLM does. It doesn't matter. But all of these are different tables of rules. However, there are certain machines, which he called universal machines, whose table of rules allows for a different table of rules to be written as symbols on the tape.

So this is a weird, like a, you know, kind of meta move in which you say, oh, there are certain machines where the rules themselves can be written as data on the tape along with the inputs. And that means that those universal machines can perform any computation, not just the computation that their table of rules tells them to do. So, you know, what I've just described is exactly what a computer is. A computer is a universal Turing machine. And when we say that it follows an algorithm or runs code, the code is exactly that written out table of rules, you know, that is actually written in the memory and that is read by the machine. So it's a really interesting move, and it's one that allows for arbitrary behavior to be done by a system that is itself quite simple and has fixed complexity.

Now, if we just think Turing machines generally, meaning not universal Turing machines, but just any old Turing machine, all that is is something that is abstract, that has a state, that can take inputs, change its state, and produce outputs. Now, that is such a general construction that it applies to anything in nature that you want to think of that way. It could apply to Jupiter. It could to the water in a stream if we define state input and output in the appropriate way. But a universal machine, in other words, one where the directions that it's following define the things that it does, that's something a lot more particular. And that really begins with life.

I should also say that there is a key concept here. It's a little bit deep mathematically, but it has to do with the mapping from microstates to macrostates. The reason that time has an arrow, that the past is the past and the future is the future, has everything to do with entropy, with the fact that things go from more ordered to less ordered states. So if, for instance, you're looking at a bunch of balls bouncing around on a pool table, but suddenly you see them all come together into a triangle and one ball shoot away with all of the momentum, now you know you're watching the movie in reverse.

And it turns out that computation and entropy are also very closely related and that you can think about computation as really being the story of how causality comes into the world. And when you look at any computer language that's ever been invented or discovered, they always have something like an if-then in them. And the reason that they have an if-then is because they need to be able to create causes in the world. Anything that is a behavior, that is a computation, requires conditionality. that you do something on the basis of what was input.

Even if you think about a thermostat, when the temperature drops below a certain level, it will turn on the heat. And of course, the heat will then cause the temperature to rise, which will then cause the switch to turn off. So that's an if-then. You know, it's a minimal form of behavior. The behavior of a thermostat is conditional on the temperature. So, you know, that's sort of the elementary building block behind anything that can be said to behave. And it's also an elementary building block in anything that can be said to compute. The science of causality is a way to think about computation and how computation arose in nature, and especially in the emergence of life, as opposed to just being about computer technology that we began making in 1945.

Now, there's another really interesting chapter to the story of computing itself, which was really invented by John von Neumann. Now, John von Neumann was trying to work out the theory of how it is that a thing can reproduce. He was trying to solve the Baron von Munchausen problem, you know, pulling yourself up by your own bootstraps. How could life bootstrap itself when you need to have an instruction tape in life and that instruction tape needs to have the instructions for building a self? But, you know, if the self is copying the instruction tape, then, you know, it would seem that you have to have an infinite regress. You know, in order for you to be able to construct yourself, you need to kind of have a copy of a copy of a copy of a copy of yourself. And this is kind of like a logical paradox.

So the way von Neumann solved this was by saying, well, if your inner instruction tape has the instructions for making two different machines, as he called them, machine A and machine B, machine A would follow the instructions on the tape in order to build whatever it says, and machine B makes a copy of the tape. In other words, it treats those instructions as symbols rather than treating them as instructions. And if the instructions for building both machine A and machine B are themselves also on the tape, then you kind of break the paradox. You have a system that can reproduce itself.

This is really cool because he figured it out in the late 1940s. And this was before Watson, Crick, and uncredited colleagues had figured out the structure and function of DNA, which is exactly this instruction tape that he was talking about. But also before the discovery of the ribosome, which is what he called machine A, or the discovery of DNA polymerase, which is what he called machine B, both of which indeed have the instructions for building them in our DNA. So he got all of this exactly right. And also, he realized that this was basically the recipe for a universal Turing machine. In other words, a computer. Machine A literally is following the directions on a tape in order to perform certain actions. And that's basically how Turing defined computation.

"The birth of biology is the origin of the first computers."

So what this means is that the original symbiosis that gives rise to an organism that can reproduce itself heritably is literally the coming together of a computer. So in that sense, biology is computer science. The birth of biology is the origin of the first computers. And what we have today in our phones, our laptops, and so on, we should call artificial computers in the same way that we call AI systems artificial intelligence.

We know from von Neumann that life has to be self-constructing. In other words, it's got to be able to grow, heal, and or reproduce. Otherwise, it will no longer exist in the future. If it's complex, if it's more than just a really trivial crystal, then that requires general computation in order to build itself. And in particular, you need the heritability property. In other words, you need it to be the case that it's got a set of instructions inside itself such that if those instructions are changed, then future generations will also change how they are. Right. So there's got to be this kind of persistent instructions that allow heritability to occur. So life is general computation, or I called it playfully computronium, meaning matter that computes.

And the other part of that definition is that, you know, once you've got that property, if what it's interacting with is other life, which will always be the case, by the way, you know, because the moment life can reproduce, suddenly it's in a bath of other life, then that opens the possibility for something called symbiosis.

Darwin was always troubled by the question of how life gets started. We tend to assume that life evolves through random mutation and selection for fitness. So this is called the neo-Darwinian synthesis. It's not exactly what Darwin came up with, but it's what I guess Darwin settled into as he was interpreted by later thinkers. So the idea here is that we have a genome, and that genome gets hit with radiation, randomly mutated, and those random changes stick to the degree that those random changes confer some survival advantage. Now, that story is not exactly wrong, but it's only half the story.

As it turns out, the big steps in the advancement of life have come from cooperation, not competition. They've come from entities, two or more entities, sometimes many entities, with either the same genome or different genomes coming together in an act called symbiosis. Symbiogenesis is when two or more life forms come together to make a more complex life form that itself can reproduce. If it cooperates with the life around it, then it can become obligate, meaning that those cooperating things have to now reproduce together.

Now, the story of symbiosis, which is to say that things come together, cooperate to make bigger, more complex things, turns out not only to address the problem of how life becomes becomes more complex over time, but also the problem of how life gets started. So it turns out that when molecules cooperate, meaning molecules somehow end up in relationships with each other, chemical reactions, whereby those reactions lead to more of those molecules, and then those cycles of chemical reactions, which are autocatalytic, would be the technical term, start to interact. Those can make systems that in turn propagate themselves. That story of symbiosis going all the way down to molecules is not only the story of how life becomes more complex, but also of how life arises in the first place.

And this story of cooperation is not what we generally think of when we think of evolution. We think only of competition. But cooperation is how we went from molecules to bacteria. It's how we go from bacteria to eukaryotes, the kind of cells that we're made out of. It's how we go from eukaryotes to multicellular organisms like us. And it's also how we've gone from individual humans to societies. And that cooperative story is just as important as competition. In fact, in some sense, cooperation and competition are not even distinct. They're parts of the same larger story.

Lynn Margulis and the major evolutionary transitions

The idea of symbiosis dates back at least to the beginning of the 20th century. But the person who really brought it into the mainstream and proved that it was true, that it really had happened, was Lynn Margulis, who was a real kind of rebel biologist. The paper in which she described this, it's from the late 60s, was rejected by her account from more than a dozen journals before it was finally accepted in the Journal of Theoretical Biology. And it was very controversial for many, many She eventually won the National Medal of Science in 1999, but it remained a very controversial story right through most of the 20th century. By now, it's accepted by pretty much all biologists.

And what Margulis and others before her had noticed is that mitochondria look a lot like certain kinds of bacteria that we can find free swimming in the world. They also, as it turns out, have their own DNA. So there are little loops of DNA inside the mitochondria, and those loops look kind of like bacterial DNA. And what Margulis showed was that, you know, indeed, there was a common lineage between mitochondria and free-swimming bacteria that are out in the world. But she believed that symbiosis, this fusion of simpler life forms into more complex ones, was not just something that had happened with mitochondria, where she proved it, but was the engine behind. All biological development, all biological complexification. And that is still not mainstream among a lot of biologists.

However, in the 1990s, there were two, I would say, very prominent, more mainstream biologists, Eörs Szathmáry and John Maynard Smith, who at least formulated this idea that there had been some major events in evolution, including the formation of cells, including the formation of societies and hives of insects and so on. They called them major evolutionary transitions that were clearly symbiogenetic. In Szathmáry and Smith's original paper on the major evolutionary transitions, they listed eight major transitions. So, you know, we go from Lynn Margulis to John Maynard Smith and Eörs Szathmáry.

And my perspective is more like Lynn Margulis's, that those big events are extremely significant in the history of our planet, but that symbiosis is actually happening all the time and is the engine that is kind of ongoing and that is building life. Life has a selection process that is more like classical Darwinian evolution, but it also has a revolutionary process where things are changing in ways that involve recombinations of things that are more active. So this is where I say complexifies through symbiosis. So life is matter that means. It's matter that has folded itself into a computational structure that can replicate itself, that can construct itself, and that in turn cooperate with other matter that has done the same thing in order to make more complex life.

And we see evidence for those kinds of symbiogenetic events, you know, which are not the big ones that Smith and Szathmáry identified in stuff like DNA. And I'll use human DNA as my example here. So, you know, we first sequenced a full human genome back in 2001. And one of the big surprises in the human genome. is that only about 1.5% of it codes for the proteins that make up our own bodies. So what is the rest of it? This is often called junk DNA. It's not really junk. A lot of it is involved in regulation and in other functions that we don't actually fully understand.

But when you start to explore all of that sort of 98% of the remaining DNA, what you find is that a whole lot of that so-called junk DNA is actually the remnants of retroviruses that appear to have injected pieces of their genome into our genome and moreover into our germline, the part that reproduces from generation to generation. And so the HIV virus is an example of one of those retroviruses. Now, what HIV doesn't do is to insert itself into the DNA of eggs and sperm. But if it did do that, then not only would it infect you, it would infect all of your offspring as well. Which is kind of a creepy thought. But when you look at what our DNA is made out of, most of it appears to have been made out of DNA that had exactly that historical evolutionary trajectory, that it was injected in by other things. So in other words, our own DNA is this kind of overlay or palimpsest of the DNA of all kinds of other entities that reproduce inside DNA that have, in some cases, jumped between species. and so on.

And actually, we have a lot of evidence that those pieces of DNA, which might originally have been antagonistic or might have been infectious, have taken on critical for us. So for instance, there is a virus called ARC, A-R-C, which is endemic to all mammals. And it turns out if you knock it out in mice, those mice stop being able to form new memories. So somehow this piece of viral DNA, which still forms virus-like structures in our own bodies, is intimately involved in memory formation in our brains, which is kind of wild.

Or there's another protein that fuses together cell membranes. It's a little bit like the proteins in RSV, which is a respiratory infection that infants and some older folks get as well. And that fusing together of membranes turns out to be what forms the placenta. The placental barrier is formed by fusing together membranes of adjacent cells to make a kind of barrier. And that also looks like it was the product of some kind of retroviral invasion a long time ago.

So, you know, when you start to look at those sorts of events, actually life stops looking like a tree at all. It starts looking more like a tangled bush that is always kind of fusing and coming back together with itself as well as diverging. And so as we start to understand more about how evolution works, that these cooperative symbioses are really important, that it doesn't work like a tree, but more like a bush that intersects with itself, that fuses all the time, things come together all the time, it changes all sorts of things of things.

So one assumption is that organisms are passive and mutation is some random phenomenon. That seems not to be the case, you know, since a whole lot of the changes are active. They involve things, you know, literally injecting themselves into other things or the sharing of code, you know, back and forth. It's more like, you know, like an open source code repository like GitHub, where people are constantly, you know, importing libraries and sharing code with each other. You know, the code of life works like that too.

And that actually implies a lot more agency on the part of life itself as well. There's a great historian of science, Jessica Riskin at Stanford, who has written about this. She actually just wrote a new biography of Lamarck, who we all learned was wrong in high school because he believed that giraffes have tall necks because they stretch their necks up over a revolutionary time to reach the high parts of trees and they self-constructed. And of course, we know that that literal form of self-construction is not true. The length of a giraffe neck is determined by its DNA. But the deeper point that Lamarck was making is that he believed that life constructs itself, that life has a lot of agency in evolution.

And the contrary view was actually a religious view that living stuff has no agency because all of that agency belongs to God. And so in that sense, at least everything that is non-human doesn't have any kind of will. It's purely acted on by external forces. Evolution just selects for whatever is fittest. And that's not the case, right? If you look at genes infecting each other, at genomes that are combinations of other genomes, at things that actively edit themselves, it's a much more active story. And it's a story that also includes a lot of elements of cooperation as opposed to competition.

Another thing that I think comes to the fore is that when you have entities that cooperate to form a larger entity, when you have a symbiogenetic event, suddenly the game that is being played changes. The roster of players changes and new evolutionary niches open up, new possibilities for play, if you like, open up. And what that means is that this idea of fitness, of there being a fixed fitness landscape and everything optimizing on that fitness landscape is also wrong. Because it turns out that the landscape moves along with the players. You know, all of the players are each other's landscape, if you like, and the players are constantly changing because of their shifting cooperations and fusions and mergers.

So all of these assumptions about Darwinism, that there is such a thing as fitness in any fixed sense, that you can rank two entities based on which is fitter than the other. You can't because it depends on their environment and on the context and on how they're cooperating or competing with each other at that moment. The idea that organisms have no agency and are just acted on from without and that evolution doesn't have a guiding hand, if you like. No, it has many guiding hands. It's those organisms themselves that supply the guiding hands. All of those assumptions are wrong. And I think that they've infected many other fields as well, everything from game theory to economics. There's, of course, social Darwinism, which we know leads to all kinds of pretty ugly conclusions. All of those are not only morally wrong, they're actually wrong from a scientific perspective as well, because they're based on a mistaken idea about what evolution actually is and how it works.

So, yeah, life is self-constructing computation that complexifies through symbiosis. Now, I'll bring this to its logical spicy place, which is, is AI alive? I've mentioned the two, in my opinion, big founding fathers of computer science, Alan Turing and John von Neumann. There is a difference in the way they thought about computation that I think is important. So for Turing, the Turing machine is an abstract thing. It's an abstract head that moves left and right along a tape and reads and writes symbols on that tape. But it doesn't actually have the ability to read and write. A copy of itself. It's not self-modifying. Whereas for von Neumann, this universal constructor is something that can literally build another universal constructor. In other words, what it's reading and writing is atoms, not just symbols.

And that's important because, you know, if you think about a phone, for instance, it's a universal Turing machine in the sense that it can compute anything, but it can't pop another baby phone out the bottom, you know, the way half of us. can. So that ability to literally make a copy of yourself or do what all of us can do, which is to grow and heal, that's also a form of self-construction. That's something that these universal constructor-based von Neumann machines can do, but the Turing machine can't do. And that's important because, you know, life is not just abstract computation that involves, you know, symbols in some abstract mathematical space. It's, you know, literally the reading and writing of the things that you are made out of into the world.

But I should also point out that, you know, if we look around at the number of phones in the world, there are clearly more phones this year than there were last year. So, you know, where are they coming from? There's a kind of silly quote from an astrophysicist, Robert Jastrow, in the early 80s. He said that we are the reproductive organs of computers. I think, you know, in some sense, that's correct. You know, AI, in a similar sense, doesn't exactly create more AI in isolation today. So it doesn't have that, you know, that basic self-constructing property. It's all made on Turing machines, not on von Neumann machines. The things that computer programs, including the ones that run AI models, read and write are digital symbols, not actual. computers, right? Not the wires and silicon that make up the actual physical basis of computers.

However, we are certainly making a lot of computers nowadays. There's a lot of data centers getting built. So, you know, when we look at the system more broadly, you know, you can think about AI as being extra brain tissue that is being made by a larger living system, which is, you know, us and our entire technosphere. And in that sense, steam engines, computers, AI, they're all part of the same big story. You know, this idea that we are somehow separate from nature, that everything that's been going on in the last few hundred years is unnatural, I think is another of those fallacies that comes from misunderstanding how nature works in the first place. This is actually part of the same story.

It's kind of a matter of which lens or which perspective you look at it through. You know, in the same sense that you can look at an insect colony and you can see a superorganism made out of insects, or you can see a bunch of insects. You know, in the same way, you can look at a eukaryotic cell and you can see either one entity or many entities, all of these, you know, bacteria on the inside, which are the mitochondria, cooperating with the bacterium on the outside, the big cellular membrane that they're all inside of. And both of those perspectives are true.

So, you know, symbiosis in a way is about naming the assemblies that things coalesce into as new things onto themselves that are more complex in terms of how they function because of a sort of division of labor of all of the parts. That's sort of a general recipe for symbiosis. And when you look at it that way, the story of human society over the last 10,000 years just looks like another of those symbiogenetic events. You know, yes, we're all still individuals, but also we are something bigger than that when we come together and we're capable of so much more when we come together in just the same way that bacteria coming together into a eukaryotic cell are capable of something much more complex than individual bacteria are.

So the thing that characterizes living systems as opposed to non-living ones is function. Functionalism is the idea that things are what they do for other things, if you like, that everything is about relationships as opposed to being about some inner essence. If you think, for instance, about a rock. If you break a rock in half, we don't say, oh, the rock is broken now. It doesn't work. A rock broken in half is now two rocks. Whereas if you break a kidney in half, you now no longer have a kidney. It no longer will function. And that's a property that is true of life, of living systems. They have purpose, they have function, as opposed to just being matter.

And what does that mean? What it means to have purpose or function is basically to be in symbiosis with other things that also have purpose or function and whose inputs and outputs somehow are hooked up to yours. So, you know, your outputs might be their inputs and vice versa.

And Alan Turing, who was one of the founders of computer science, he was a mathematician. So he was thinking about mathematical functions. If the function of some system is, say, to add numbers together, then his point was that you could make a machine, a physical machine that would do that out of cogs, out of electrons that move around on wires, out of people waving flags. It didn't matter as long as the right transformations were happening in the information in that system. So that idea that it's the information that matters and the substrate can vary is something that he called substrate independence.

And this is really important because, you know, if you have something like a kidney, when we think about what a kidney is, it's really functional. Its job is to take the urea out of blood so that the rest of your body survives because too much urea accumulates in the blood. It's toxic. So we're able to build something like a dialysis machine in order to substitute for a kidney precisely because it's the function of the kidney that matters, not what it's made out of. Now, obviously, you know, a dialysis machine isn't as good as a kidney in a variety of ways. You know, nobody who has the choice will choose a dialysis machine instead of a kidney. But the fact that it keeps you alive is kind of passing the kidney Turing test, if you want to think about it that way. It's saying that when you substitute something that has the same function, then it works the same way as the original thing.

And nature has played this trick many, many times. You know, when you look at how different functions are carried out in biological systems, you see that nature is a great inventor. Flying, for instance, has been invented a number of times in nature, all of which are made very differently, but they all serve the same function. They all fly. So, in other words, there is something profoundly ecological about the idea of function and profoundly symbiotic or symbiogenetic that function is defined relative to other functions and functions work insofar as they work with others.

So, for instance, I kind of need to wear glasses, although I usually don't nowadays. But the fact that the lens is made out of glass as opposed to out of the transparent cellular material that our own biological lenses are made out of is not the important thing. The important thing is that they refract the light in a certain way. And seeing is all about that light getting refracted the right way. It doesn't matter what it's made out of. If we substituted the glass for quartz of some other kind or for plastic or what have you, it wouldn't matter. These might have different properties in terms of their weight, in terms of how well that ecology works on the tops of your ears or your nose, but it doesn't matter as far as vision is concerned.

So this idea of substitutability, of one substrate or one pathway being able to take the place of a different one is something that nature uses all the time. as well. When you look, for instance, at metabolism inside a cell, the point is to make a molecule called ATP that carries energy, and we have multiple chemical pathways in cells for doing that. So, you know, In some sense, biology is the origin of function. It's kind of an ecology of functions that work together to make bigger functions. And the platform independence, in other words, the fact that you can substitute different materials, different processes for how those different functions work, is fundamental to how life works. Because it's all about the relationships, not about how it's done or what's inside.

And I'll bring this to its logical spicy place, which is that a lot of people will, you know, are still saying things like, you know, AI is just running on a computer, like a computer can't be intelligent. That's a property of brains and, you know, living systems. And. And I guess, you know, the reason that that doesn't strike me as correct is because, you know, the fact that a system is living is precisely the fact that it is functional, that it serves these ecological relations with other things. You know, when neurons are talking with each other with their electrical signals as they do and chemical signals, they absolutely are substitutable for other kinds of entities that are doing that same kind of interaction with each other.

There's a thought experiment. It's been articulated by a couple of philosophers, David Chalmers, Susan Schneider. They call it the brain of Theseus by analogy with the ship of Theseus. The ship of Theseus story being if there's a ship and you start to replace its timbers, does it become not the same ship at some point when all the timbers are now different? And the answer, I think, is no. By any reasonable definition, if you replace the timbers continuously, it's still the same ship. That's certainly the case with us. All of our molecules are replaced as we go through life. It's still us. So we are literally the ship of Theseus.

But their version of this is a brain of Theseus, where you imagine taking a neuron. And replacing it with a computer that has the same inputs and outputs. And, you know, we kind of know in theory how one might do that. Brain-computer interfaces are very difficult to make, but we sort of know how individual neurons compute. That's been sort of worked out since the 1950s. And so, you know, the question is, if you replaced one neuron in your brain with a computer wired to the inputs and outputs, Would you feel any different? And, you know, I think the answer is pretty obviously no. But then, well, what if it's two neurons? What if it's 100 neurons? What if it's a million neurons? What if it's all of them? You know, would there be some moment at which, you know, you would suddenly, your consciousness would fade out or you would stop, you know, being truly intelligent or whatever? And for me, I think the answer is very clearly no.

"Because the you is in that ecology, in those relationships, not in the particular molecules, not in the particular cells."

You know, if you did that perfectly, if you maintained all of those relationships, all of those input-output relationships, then nothing would feel any different. Because the you is in that ecology, in those relationships, not in the particular molecules, not in the particular cells. It's the organization of them, and it's not some magical property of those molecules. of the matter. So, you know, that's functionalism in a nutshell.

And to me, you know, if you sort of pair that together with Turing's big insight about, you know, this being essentially computation, you know, behavior is something that can be expressed in terms of computation, there's not any very profound difference between, you know, between computation happening on a silicon substrate and computation happening in a neural substrate. There are certainly differences in terms of the efficiency. Biology is still in some ways much more efficient than silicon. It's actually one of the things that my team is working on today. And also, by the way, reliability, which kind of goes along with efficiency. There's a precision about the way we compute in silicon that's really a function of the fact that we use the same kinds of chips to calculate spreadsheets and stuff. And we don't want anything to be off by a little bit when we do a sum in a spreadsheet. It's probably an error to be bringing that same kind of precision to neural computation, by the way.

But if we kind of... Defocus our vision a little bit and forget about those details. And we just think about that ecology of relationships. Whether you do it in silicon, whether you do it in proteins, doesn't matter. For example, in the movie, The Hail Mary, there's an encounter with an alien who is made out of kind of mineral or rock. And, you know, Of course, the substrate is totally different, right? Rocky doesn't have neurons in the same way that we do, isn't made out of proteins in the same way that we do. But, you know, most of us would not say, well, therefore, Rocky can't be having real thoughts. You know, therefore, He can't be conscious. Of course not. And in the same way, I think that's true of computer systems, of any other substrate. It could have been done with Babbage's steampunk computer that relied on cogs and wheels and belts. And the whole point is that when you look at things from a functional standpoint, it really doesn't make any difference.

So the thing that life has that non-life does not is function or purpose. And in particular, the parts of a living thing have purposes that they serve for each other. You know, the heart is there to pump blood. The lungs are there to breathe air and to, you know, put. Oxygen into the blood and get the carbon dioxide out. Kidneys are there to get the urea out and so on. So, you know, all of those things, functions or purposes, are what make life, life.

And this is also something that, you know, that a lot of scientists have struggled with because there was this idea in the 19th century that we should get rid of any idea about teleology or purpose in nature. That, you know, nothing has any purpose in nature. It's just atoms and molecules. It's physics. It's chemistry. You know, this idea that purpose doesn't exist in nature was kind of seen as a way of trying to remove religion. From nature, you know, or remove Aristotle's idea that air wants to rise or something like this. But it actually, I think, had a really perverse consequence. It's kind of like trying to get the lump out of a carpet by flattening it. And what you end up doing is making a really big lump in one spot because you've tried to iron the rest of it flat.

When you try to get rid of purpose, what you end up doing is essentially moving all of the purpose into one spot. And, you know, in the case of the 19th century biologists, that one spot was God. Or human purposes, human will, which was seen as a reflection of God's will, a little piece of the divine, that we all have souls and that's what gives us purpose. No scientist, of course, would have accepted that, but we kind of moved this idea of purposes outside the purview of science and thereby either we're pretending that purposes don't exist or that they come from outside science and are supernatural. When, of course, the reality is that whenever you have living systems, they're always serving functions for each other.

And in real life, of course, if you're a biologist, you can't talk about something like the immune system without talking about its function, about what its purpose is, what it does. It's there to defend you against bacteria and viruses that would otherwise kill you. You can't talk about the function of the hairs in your nose without... without talking about why they're there, what they're doing for us, right? They're there for filtering out certain crud before it can get into our lungs, and so on. So this idea of functionalism and of the fact that functions can be served in various different ways by different kinds of mechanisms, That's the very essence of life. And life is the emergence not only of causality, of one thing causing another, but also of purpose in that causality.

I should mention here that we've actually done some really cool work on my team in the last year or so in artificial life, which artificial life, by analogy with artificial intelligence, is setting up. Sort of proto-biological systems in a computer. And this experiment is one that really investigates abiogenesis, which is to say the emergence of life out of non-life.

Here's how it works. We begin with very, very short random programs. So they're only four instructions long, four bytes long, and those four bytes are random. The idea of a random program might seem quite paradoxical, but if you look at what a program is made out of, it's just made out of instructions, one after another. And in this very simple language, those instructions are exactly one byte. So, you know, when I say they start off as random programs, all I mean is random bytes. that do something. And where do I get off with the idea that they do something? Well, molecules do something as well when they interact. If you have two molecules that interact and one of them Phosphorylates the other, that's doing something, you know, or transfer, you know, transfers an ion or, you know, breaks off a hydrogen or whatever it is, right? So all chemical reactions are, you know, one molecule doing something to another.

And so that's all it is. And the reason that I used a minimal programming language for this was just to ensure that the set of instructions that I was working with were sufficient to, in principle, allow any kind of program to work. In this case, I based the first version of this on a computer language called Brainfuck. It wasn't named by me. This was named by a Swiss grad student in the 90s who invented it. So his idea was to invent a computer language that was really minimal and that had a minimal number of instructions. And he called it that because... It is a bit of a brain fuck to try and program, and it's very difficult to program in. But anyway, the reason that I used it was because of its minimalism.

And so you start off with very short, random tapes that don't do anything. But this particular variation of the language is one where the instructions can self-modify. In other words, they can modify the tapes themselves. But in the beginning, they don't do anything interesting because they're just random instructions, right? It's like a chemical soup of just random molecules bumping around together.

The procedure is as follows. You have a soup of, let's say, a thousand of these little tiny random tapes. You take two of them out of the soup at random, and you stick them end-to-end, and you run them. So this is sort of an interaction between two molecules bumping around in that soup. And then once you've run them, you pull them back apart, and you put them back in the soup, and then you just do it again. You mix and you do it again. And you just do that over and over. This doesn't seem like a very promising way to make anything interesting happen. You know, take a bunch of really short random programs and, you know, that have the potential to read and write to themselves, but, you know, they're literally not doing anything of any interest in the beginning and just do it over and over. What do you expect to happen? Nothing.

But what does happen is actually really remarkable. So after a few million interactions, suddenly you begin to see these tapes becoming longer and more complex, and they start to really compute. A lot of computation starts to happen, and they start to reproduce. Why? Well, if you think about it, a program that replicates is going to out-compete a program that doesn't replicate. If you have a program that can replicate, it will exist in the future. Whereas if you have a bit of code that can't replicate, it will eventually get overwritten by a program that can. So in some very trivial way, evolution is already at work with respect to even non-functional programs in the beginning because anything that can replicate itself is going to persist into the future, whereas something that can't won't. So, you know, It's a sort of model for how life arose, and it shows you that life can arise actually very easily in this minimal form.

And I should also point out that the fact that these tapes began very short, but that you stick them together in order to run them is really important to this experiment, because what that allows to happen is symbiosis. It allows separate tapes to come together to make a longer tape. And that's essential in this case because the shortest program that can reproduce is actually longer than four bytes. It's more like six bytes or so. So the fact that you're putting tapes together is necessary in order to get things that are long enough and complex enough they can reproduce. And once they're reproducing, you find that those in turn... Come together to make even longer and more complex states that themselves can reproduce. So you can actually see symbiosis leading to life and then complexifying that life, even in this very, very simple artificial system.

And when you look at a BFF soup, after it's been running for a few million interactions, you find programs in it that are definitely purposive, that look like they were made by an engineer. I mean, they look like they were made by a mad engineer because they're not programmed in the way that you rationally would as a programmer, but they certainly are doing something purposeful. And if you break them, if you change some byte at random, you will generally destroy their ability to function and they won't be able to replicate. And that modified one, that destroyed one will get overwritten by something else.

So it's really what is so magical about seeing this process occur is that you start off without any You start off with just bytes, and then after a while, through nothing but random interactions, there's no externally imposed fitness function, there's no judge, there's no designer, there's no programmer. But after a few million interactions, you see programs that are purposeful, that look as if they were engineered. And all of that has just happened through interactions between things that, you know, when they're lucky, they serve a function for each other, if that makes sense. Right. So when, you know, when a couple of instructions come together that can help each other to continue to exist in the future. Then. They will continue to exist in the future. And that's the key, that's the secret behind symbiosis.

And as those pieces become more and more complex, of course, they also become more and more functional. What they do is more and more sophisticated and more and more contingent on everything else that is in the soup as well. So, you know, you can really see how not only does, you know, can life truly emerge from nothing, which is the thing that had really bothered Darwin, but you could also see how and why it becomes more complex over time.

So a simple BFF program just replicates itself. It doesn't have any more complex behaviors. But we've run some experiments in which we require those programs to, you know, do a little more work to earn their keep. So, for instance, you know, it might be the case that they need to chase a light, which is actually giving them the energy to do computation in the first place. And by the way, you know, I'm not sure I've mentioned this, but computation requires energy precisely because it is an entropy lowering process. It sort of erases information from the world. And in the process of doing that, you need to feed it free energy. That's why plants need to photosynthesize. It's why we need to eat high energy foods like fats and sugars.

And so it has to have behaviors that are about more than just copying its own matter. And those behaviors might involve chasing certain chemicals, fleeing negative conditions that are going to destroy it, like heat too high or something along those lines. So all of that requires that the computation become more complex, more sophisticated. Also, of course, one of the great sources of energy, if these little guys need energy to compute and they can store it, is other little guys. So you've got predator-prey relationships. Those kind of emerge naturally in any system that has life. So the moment you've got stuff that wants to eat other stuff, then the stuff that is potentially going to get eaten needs to try and escape as well.

So both because... because of cooperating with other life and because of competing with or even eating other life, life needs to start to model its environment and model other life. And by model, what I mean is be able to behave in ways that are going to have positive effects for your own future, given what you expect the environment and others will do, conditional on what you do. So this kind of sets up a sort of arms race of intelligence in which the more I'm able to model, the more I'm able to predict what others will do under various circumstances, depending on what I do, the better I will be able to survive.

I should mention here that we've actually done some really cool work on my team in the last year or so that shows that in a multi-agent system, so this is machine learning work, in a multi-agent system where every entity is able to model itself and model others. As part of its model of the environment, those systems are able to learn how to cooperate in ways that they would not be able to if they weren't doing this kind of self-modeling and modeling of others. So you've actually got to be able to not only think conditionally about how the environment will react to things that you do, but also how you will react to those changes. And if there are others in the environment that are like you, You've got to be able to predict their behaviors, keeping in mind that they are going to be like you as well, that they're similar to you. So all of that comes together to basically create evolutionary pressure for entities that are smarter and smarter because, you know, your ability to model others is going to be increased by the complexity of your model and your ability to learn. But the more you learn, the harder you will be to model as well. So there's a kind of, you know, friendly and unfriendly both arms race of intelligence that arises naturally whenever you get living systems.

When we were kids, we all knew what artificial intelligence meant. There was Rosie, the housemaid robot in the Jetsons, and there was HAL 9000, and the Star Trek computer, and all kinds of robots or AI systems, embodied or disembodied, that were intelligent. And that was what AI was. Now, sometime in the 90s, the term artificial intelligence started to get used for systems that were very narrow in what they could do. Things like digit recognizers. They could look at a bunch of pixels and distinguish zero through nine or distinguish between a fixed set of visual objects. Is this an apple, a pear, a Siamese cat, whatever.

And that was, I think a lot of people felt like that was an abuse of the term artificial intelligence. Certainly it required that we come up with some way of distinguishing digit recognizers from Rosie the robot. And that was why we invented this distinction between artificial narrow intelligence, meaning a system that was designed for doing a specific task, and artificial. artificial general intelligence, meaning when you could just have a conversation with about anything. So that was the original purpose of the split. I should also say that the reason that artificial narrow intelligence was called artificial intelligence at all is because a lot of it was neural network based. There had been many attempts to do things like distinguish digits using handwritten programs, and those attempts didn't work. So they were neural in some way. They were brain-like in some way, but they were very specific in what they could do.

Around 2020, a little bit before, we started to have language systems, so large language models, that were based on pre-training with a very large amount of text from the internet. And it turned out that you could start to have conversations with those systems about... Arbitrary stuff. You could ask them to write poetry or critique some prose or think with you about something, tell jokes, whatever it is. And they weren't specifically trained to do any of that stuff. They were just trained to model language. In my mind, as soon as we started to get systems of that sort, they were no longer narrow intelligence.

You know, I remember seeing John Oliver saying, you know, don't worry, these systems are still just generating text. That's not general intelligence. It's just text. To me, that doesn't make any sense. whatsoever. The whole point of text is that it is general. You can read and write about anything. This whole conversation that you and I are having, most of it could be captured by the transcript, by just text. So text is not narrow. Text is general.

And in my opinion, if you took any of today's AI systems based on large language models and you transported them back in time to the year 2000, when that distinction between artificial narrow intelligence and artificial general intelligence was first being made, I think anybody back then would have said, yeah, you're there. You've got it. This is what we meant by artificial general intelligence, by AGI. I wrote a piece together with a colleague, Peter Norvig, where we said artificial general intelligence is already here. And I certainly felt that that was easily true in 2022, even more true now. And I think we crossed that threshold when we stopped training those systems to do specific jobs and started training them on language.

So, you know, to be honest, I've actually been really confused by why it is that we are still, you know, waiting for Godot. You know, like, why are we still saying, you know, artificial general intelligence is out there in the future? What do we even mean by that? It seems like we've quietly shifted the goalposts. And maybe what we mean now is artificial superintelligence, meaning, you know, systems that are as good or better than humans at every possible kind of task.

It's true that sometimes LLMs do weird stuff. You know, they will hallucinate something strange, you know, or they'll fail to understand something, you know, that seems pretty basic, right? Or to follow an instruction correctly or something. But if we were having those interactions with Rocky Rocky the Alien and Hail Mary, you know, the conclusion would not be, oh, well, clearly Rocky is not actually intelligent. You know, it's more like, oh, you know, there's some funny gaps, you know, both superpowers and gaps here that are a little different from ours. And that should be no surprise. I mean, the... Both the way our brains are made and our developmental trajectory and the way our culture works and the way we're brought up, it's all really, really different from the way we pre-train and post-train AI models. So it would be really surprising if everything were the same, if all of the strengths and weaknesses were the same.

But it's part of maybe our nature as humans to have this sort of... anthropocentrism. And we think about our strengths and weaknesses as being the gold standard. This is what it means to be intelligent, to be whole, to be what we are. And any deviation from that must mean that it's not the real thing or something. That's, of course, something that falls apart pretty easily when you look at it more closely. I mean, for one thing, there's a huge variety, actually, of human intelligence. I mean, you know, it's not like all of us have the same abilities and capabilities either. And we also, you know, make a lot of mistakes and give each other a lot of grace for those mistakes in ways that we absolutely don't give grace to an LLM. It's like we're waiting for a guy. gotcha, you know, whenever they get something wrong. But whenever we get something wrong as humans, it's sort of like, oh, well, it's just, you know, like, maybe you just didn't have coffee or you misheard me or, you know, we overlook it.

I had a debate about this on stage with a good friend, Melanie Mitchell, and she talked about AI models not being able to get the right answers to certain kinds of simple logic puzzles. And I knew she was going to do this. So I had actually made up a couple of those logic puzzles, and I gave her one on stage, and she got it wrong. And, you know, this doesn't mean that Melanie isn't intelligent, right? And Melanie is brilliant. But, you know, of course, we all get things wrong now and then for whatever reason.

Or, you know, maybe another example would be useful. Our team was one of the first to test LLMs on medical diagnosis. And when we first did this, we found that the models would diagnose things correctly only, I don't know, half the time or something like this. And the conclusion from the team at that point was, well, I guess this isn't ready for prime time. These things are not ready to be doctors. But, you know, when we went and did the human tests, the human trials of like, well, how often do doctors get diagnoses right on the basis of, say, a description, you know, of all of the symptoms and so on? It turns out that, you know, doctors get things wrong a lot. as well. And actually, the models were already doing slightly better than doctors in general. So, you know, that doesn't mean don't see doctors. It doesn't mean, you know, it does mean maybe, you know, if it's important, you might want to get a second opinion.

But, you know, intelligence doesn't mean getting everything perfect, being able to do everything right, etc. It means that you are able to solve a variety of different kinds of problems a lot better than you would be able to if you didn't have some underlying model of how the thing works that you're talking about. If it's multiplying two numbers together, it's not just that you've memorized the multiplication table, but that you understand what multiplication means, and you have something like a multiplication algorithm in your head that you can apply to a new problem. When we find that AI is able to start advancing the frontier of mathematics, for instance, and there have been some recent results in just the past few months that show that, right? Novel theorems, novel proofs. I think that's a really nice example of cases where you can't say this is just regurgitation, that it's just sort of recalling something because they've been trained with a massive amount of training data unlike humans. The very stuff that we think of as intelligence, right? Taking what we already know, taking what we already have been exposed to, putting it together in a new way to do a new thing.

So I think that restricting ourselves to thinking about AI as tools is probably not quite right. You know, a tool is something that That's no agency or that we model as not having any agency and that doesn't do any thinking. But of course, you know, AI is exactly not that, right? It's something that also thinks as the human brain thinks. It's something that also participates in that big sort of human ecology.

So when we increase the amount of intelligence in a system, we greatly increase what it can do, what its possibilities are. You know, the fact that. That human life became so much better over the period from 1850 to 2000 had a lot to do with rising human populations. There's just a lot more brains working together. And not just a lot more brains, but a lot more brains freed to do real thinking. When most people were still doing subsistence agriculture, the huge majority of their efforts were expended just on getting enough food to eat. And it was only really a small percentage of elites who could think about science and technology and art and culture and so on. And now there are so many more of us who can do those things thanks to that division of labor and thanks to all of that unleashing of new sources of energy and new resources.

So, um, You know, my wish, you know, out of this whole AI revolution is for that party to continue. And that should be all of our wishes. I mean, that's what progress is. And again, I don't think that progress is just some new invention that the economists have informed us about from 1800 onward. It's that whole evolutionary story. There is a real progress there where complexity rises, makes systems that are more and more intricate, more and more intelligent, and that have more and more degrees of freedom, that can do more and more.

I do think that part of the problem is that we have this idea of essentialism, you know, that we are what we're made out of, or there's something about us physically, embodiment, you know, etc., that is not just about what we do, not just about the behavior or about the function. And, you know, I certainly... I'm very sympathetic to the idea that bodies are really important and that embodiment is really essential to what we are. I mean, I don't want to upload myself. I don't believe that if my connectome is reproduced in a computer and you run it, that that's going to be me in any sense that I care about. I'm really attached to my body. But I also don't think that my body is somehow magical atoms or that there's anything unique about the fact that I am made out of proteins that is determinative of my ability to be intelligent.

You know, things like intelligence really are functional. They really are about behavior. They're not about essence. And if we acknowledge that they're about behavior, about contextually appropriate behavior, behavior that reflects an inner model, an understanding of the world, of the situation, then those are all things that can be tested. They're all things that can be tested functionally. And that's why we have tests in school for whether you've mastered a subject or not. We even have tests for intelligence. Now, I don't like IQ tests. I think that they're very one-dimensional, and intelligence is a lot more complicated than that. It's not something that just reduces to a number, but it is testable. Your competence at doing various kinds of things, your intelligence, even your creativity are things that can be tested. And those are exactly the kinds of tests that we've been busy making with AI models and that AI models are largely doing a pretty good job of.

I mean, in a way, I think some of this has to do with the question of agency. We think of ourselves as agents that do stuff in the world, and there's a prevailing view that AI is purely passive. It's just you give it a prompt, and it comes back with something, and it's very tool-like in that way. Nowadays, we're actually in the middle of a pretty big transition from AI models that are just prompted and returned with some stuff to agent. Agential AI systems that do go out and do a bunch of stuff and then come back. So there'll be actions taken in the world or a bunch of thinking that has taken place, even spawning of new agents. So that certainly throws a bit of a spanner in this idea that they're just tools that are purely, whose actions are purely caused by human input, if you want to think about it that way.

But also, you know, I think we should interrogate this question of agency from the perspective of relationships and the fact that we're always in interaction with all kinds of other people. We are always, if you like, being prompted by others. You know, all of our actions are, you know, both our own and they also are contextual. They're about the situation, about what we have just taken in, what others have just told us. It's kind of relationships all the way up and all the way down. Nothing happens in isolation.

So yeah, what is the purpose of an AI system? Well, basically, it's to be helpful. When you interact with an AI system, when you either spin up an agent to go and do something or you have a conversation with it, the fact that we have more AI in the world now than we did six months ago is because more of those interactions result in something interesting, useful, valuable happening for the people who are creating the platforms and putting in the energy and the time and so on that makes it useful for those entities to exist in the world. So they definitely have purposes. Those purposes interlock with our purposes. They're not really separable from our purposes. And I think in an even broader way, their intelligence is increasing. basically not all that clearly separable from our intelligence, from human intelligence. I tend to think a lot these days of human intelligence as not just being collective in the sense of being a collective property of a bunch of biological human brains, but of this whole system that includes this artificial neural tissue, if you like, as well.

And I think it's actually a little helpful to not start the story with AI, but to start it earlier, for instance, with the Industrial Revolution. If you look at the history of economics before the Industrial Revolution, before, let's say, 1750 or so, humans are stuck in a Malthusian trap where when we grow in number, when we reproduce and become larger in number, our individual wealth goes down. We become poorer and sicker. And then generally a plague comes along and kills a bunch of us. And then we reduce in number, but our wealth goes up. And we're kind of stuck in that oscillation for many, many centuries.

And then something really interesting happened, which is that we began to burn fossil fuels. Now, we know that burning fossil fuels is not necessarily the right thing to be doing for our energy going forward, but it was a pretty amazing party while it lasted. And in particular, you can think of it as sort of like the mitochondrial event. It creates a new source of energy that lets us... us greatly exceed that Malthusian boundary that we were kind of bumping up against. So we went from 1 billion people to 8 billion people largely as a function of starting to combust, to metabolize externally outside of our bodies in engines.

"Steam engines have made seven-eighths of us as well."

So, you know, I find that a very profound thing because, you know, we know that people make steam engines. But what I don't think we generally appreciate is that steam engines have made seven-eighths of us as well. This is a mutual construction sort of story in the sense that you and I probably wouldn't exist if we hadn't found collectively through that sort of intelligent symbiosis this new source of energy to harvest. Now, what that emphasizes is, you know, symbiosis is not always between things that have long histories and preexisted. You know, the steam engine was something that was invented at a particular time and that then we fused with. We could talk about glasses. We could talk about clothes. We could also talk about cognitive aids of all kinds. I mean, pencil and paper are incredibly powerful cognitive aids. Books, magazines, newspapers, you know, music, right? All of these are cognitive aids as well.

But of course, there is still a difference in kind when we look at what AI represents. It's not just something that you can relegate to the idea of a cognitive aid. It's very clear, at least to me, that however you define thinking, Thinking itself is happening in the model as well as in your head when you're having that interaction. Now, there's good and bad here. If you start to offload all of your cognition onto another system, then that's going to disable you. So we can think about kids who have the AI model do all of their homework. They're not going to learn anything. It's kind of pointless to do that. It's especially pointless if your teacher is then grading all of your papers by feeding it right back into an AI model. That's a pathological situation. So I don't want to be Pollyanna about some of the problems that this raises, some of the challenges that this raises.

On the other hand, my own experience working with these models has been incredibly empowering and kind of the opposite. When I am thinking something through and... And I work with some of the more advanced versions of these models to work out an idea. I feel like it's not just like having a jetpack on, although it does have a little bit of that property. It's like having... Literally a bigger brain. And even when I then walk away from that model, from that interaction, I feel like I am leveled up by having had all of those interactions. So that's the ideal scenario in a symbiosis, that you sort of level each other up and are able to do things together that are more than you could do separately. And that even when you are apart, right, you are more than you would be if you hadn't had that interaction. So that's sort of, you know, symbiosis at its best.

But I do also want to note that even that can create dependence. And dependence is not necessarily a bad thing. You know, you and I wouldn't be able to survive out in the woods, right, without our support system, you know, of all of our fellow humans, you know, supplying our food and shelter and all that kind of stuff. So, you know, we probably want to do some pretty serious thinking about what kinds of independence, what kinds of, you know, unaugmented abilities we want to make sure that we are preserving as individual thinkers. But at the same time, the benefits of this kind of collective symbiotic thinking with agents are also very, very obvious to me, and they're very powerful.

I think that preserving relationships, being very intentional about how you interact with others, human and non-human alike, is evergreen. That's really important. I also think that skills remain really important. The idea that we're free to de-skill ourselves in all kinds of ways because now there are others who can do it is actually not how skilling and de-skilling have worked in the past. that there's a restaurant on the corner and they make really good food, you know, actually leads to more restaurants on that same corner that make other kinds of food also. It doesn't lead to, you know, everybody just collapsing onto a monoculture of the one restaurant, right? So, you know, not thinking in monocultural terms, not thinking in bowing out sort of terms, but continuing to invest in skills and in relationships feels really important to me, even as we take full advantage of all of those interactions and all of those possibilities in other people and in AI.

So, you know, it's quite a weird and profound thing. And I think that the reason we get so confused about it is precisely because we have, you know, one of our fallacies, maybe in the West in particular, is in believing that we are somehow separate from nature. You know, and all this thinking stuff that we do is not a part of the natural world. When, of course, it is, right? There's no boundary between the natural world and us and our technologies.

So the philosophical zombie idea is something that philosophers of mind came up with in the second half of the 20th century. And the idea is to have a system that behaves just as we do, but that has no inner life, has no experience. So, you know, zombie in the sense that there is behavior, but there's no self. There's nobody at home. Now, I actually think this idea is kind of... a chimera. I don't think that it exists.

It's sort of like saying, you know, imagine that there is a system that has tabulated the answers to every multiplication problem. And so It gets the answer right to every multiplication question you can pose to it just by recalling an answer from that table. You know, it's not actually able to do multiplication for real, but it's able to pass every possible multiplication test that you could pose it. You know, even in that very simple example, right, you could think about that as like the zombie multiplier, right? That's just recalling. It's not really multiplying. It's possible to imagine that in theory, but it's actually not possible to build that in practice. A table with every multiplication problem worked out is way too big to store in any known system. And where would all that information even come from? How could it harvest it? It would have to be... You'd have to really cook up something pretty weird in order to imagine a system like that.

In practice, testing with a bunch of multiplication problems is something you can do to determine whether that system knows how to multiply or not. And I think it's the same with respect to interactions with other entities to see whether they have a theory of mind. In other words, whether they know what it is like to be themselves, whether they know what it's like to be you. And what it's like to be a you who has a theory of them and so on, up to high order. This concept of theory of mind, in other words, being able to model your own mind and being able to model the minds of others, is really important to social interaction of any kind. Whenever we work with others, we have to be able to sort of put ourselves in their shoes in order to know how they will react to different things we might say, to know what needs to be explained and what doesn't, to know how to cooperate with them in order to get something done together.

So, you know, theory of mind arises automatically whenever you have intelligences that are, you know, in the world together and are forced to cooperate. And really, you know, is there something it is like to be, you know, a large language model? You know, what we're really asking is, you know, is there a theory of mind in that entity? You know, does that entity have a sense of what it is like to be themselves? Now, you know, this gets into some pretty deep questions of consciousness. I don't think that we're going to agree. I don't think people are going to agree for a long time about whether there is consciousness in LLMs or not.

"It's that infinitely reflecting set of mirrors that is what we mean by consciousness."

My take on this, for what it's worth, is that all we have are theories of mind. In other words, consciousness is not a thing that a piece of matter has intrinsically or doesn't have. It's not some essence or spark. Rather, it is an idea. It is an attribution that that entity has of itself, a model that it has of itself. In other words, it's not just that I am conscious. that I think that I'm conscious. I have a sense of myself being conscious, and I have a sense of you being conscious. And I have a sense that you think that I think that I'm conscious. And I have a sense that you think that I think that you think that I think I'm conscious and so on. And that's all it is. It's that infinitely reflecting set of mirrors that is what we mean by consciousness.

So, you know, do LLMs have that? Well, when you work with an LLM, you have to continually model, you know, what it is like to be that LLM in that conversation in order for that to be an effective conversation. There's a fun book called Co-Intelligence by Ethan Mollick. He wrote it a few years ago just to talk about how to effectively work with an AI. And when he uses words like think or act or understand, he puts them in scare quotes in the beginning when talking about an AI thinking or understanding. But then he says, I'm going to drop the scare quotes now because in order to have an effective interaction with an AI, you really have to just treat it as you would an intern, maybe with certain somewhat strange disabilities that people don't generally have. But treat it like a person, and then you'll have the most fruitful working relationship. And I think he's correct. I think that that's exactly why persons exist.

The philosopher Dan Dennett talked a lot about the intentional stance. In other words, for him, agency had less to do with some intrinsic property than with a perspective that you take toward something. So, you know, you can treat something with the intentional stance or not. And of course, we treat each other with the intentional stance. In other words, we model each other as having agency. And that's really important to our successful collaboration.

So, you know, just to make that very practical, if I am having an interaction with an LLM and I think of it in a very tool-like way, I imagine that I'm the one who has to have all of the ideas in this thing and all it can do is execute. You know, it's sort of, you know, craft for me a, you know, a function that does the following thing or, you know, write me one paragraph of text that, you know, that expresses the following idea. Well, you know, I get out what I put in, as it were. If I instead engage it as a thought partner, and I've tried this trick. Actually, somebody on my team, Rif A. Saurous, suggested this a month ago, and I've tried it. At the end of a conversation with an LLM, just say, you know, so, you know. I'm going to now give you a bonus turn to just think about whatever you want. You know, and, you know, you can either share it with me or not, however you'd like. You know, go ahead, spend as much time as you'd like.

So often you get something really interesting out of that last turn. Because usually what they will do is reflect on what you've just had the conversation about. And frankly, almost every time that I've asked that with one of the more sophisticated reasoning models, I get back a reflection and insight about what we've just been discussing that is new to me, that's really interesting. That if it came from a person, I would say, wow, that's really creative. That's really changed my thinking about this.

So, you know, I do think that that's a really interesting moment. It challenges us in certain ways. But some of those challenges, you know, as I mentioned, are, I think, challenges of assumptions that weren't true in the first place. You know, the idea that the individual human brain, that the singular human is the locus of intelligence, neglects the fact that most of the things that we consider to be the great achievements of human intelligence are achievements that a lot of people have made together through division of intellectual labor. It's not the case that an individual person made computers or made AI.

When you take out the phone in your pocket and you consider how incredibly complex it is, the computers that I grew up with, the Commodore 64, It was essentially invented, developed by one person, although even there, of course, it relied on standing on the shoulders of other people who had made transistors and done a lot of the other fundamental inventions that it relies on. But our computers nowadays are so... complex that there is no single person that understands even a fraction of what is going on inside those. They're collective achievements.

So I think a lot of that uncanniness that we feel, we might feel a little better about, a lot of that uncanniness can kind of evaporate once we come to grips with the fact that we are already a part of something way bigger than ourselves. There's already a collective mind here that we both participate in, benefit from, are a part of. It's not as if this is the first new mind stuff that human society has invented. So I think we're participating in part in a grand story, and we're doing so with greater and greater potential and greater and greater agency. That agency is very much a part of the picture as well. So there's a lot of good there, and the trick is going to be to figure out how to get as much of the good with as little of the bad as possible.

From von Neumann's blueprint, which predated the discovery of DNA, to viral genes that now help form our memories and placentas, Blaise Agüera y Arcas argues that function defines life more than material does, and that AI belongs in the same story. He explains why he thinks AGI is already here, and gives a blueprint for using AI without sacrificing your own mind.

My name is Blaise Agüera y Arcas. I am the CTO of Technology and Society at Google and the founder of an organization there called Paradigms of Intelligence, which does basic research on AI. I also have just published a book called What is Intelligence? Today on Big Think, we're going to talk about life, functionalism, and how to live with AGI.

So after thinking about it for a while, the definition of life that I've come up with, and I should caveat this, that every biologist's definition of life is going to be different. This is not a term that there's any consensus about. We actually wrote a paper about that in which we interviewed all kinds of interdisciplinary researchers. And for 60 experts, there are 60 definitions in there. But mine, for what it's worth, is that life is self-constructing computation. That complexifies through symbiosis. So what does that mean? Let's start off with self-constructing computation.

A lot of people, when they hear computing, they have associations with a very specific technology that began in 1945. They think of it as an applied science. And computing, as Turing and von Neumann thought about it, is actually something way more fundamental. It's a basic science in the same way that thermodynamics in the 19th century, which describes phases of matter and so on, is a basic science and long predated steam engines.

So Alan Turing developed a theory of what computation is. He did something really interesting, which was to look at computation, which at the time was done entirely by humans, and define a mathematical abstraction for what it means to do this kind of computation. He stripped down the system. system to something very, very abstract, to just a read-write head that can read and write symbols on a tape called a Turing machine.

Now, the real magic in Turing's mathematical proof is that he then said, well, You know, any computation can be defined, you know, with the right table of rules. It could be doing whatever. It could be adding numbers together. It could be taking square roots. It could be, you know, figuring out the statistics of a whole corpus of text, you know, doing the kind of things that an LLM does. It doesn't matter. But all of these are different tables of rules. However, there are certain machines, which he called universal machines, whose table of rules allows for a different table of rules to be written as symbols on the tape.

So this is a weird, like a, you know, kind of meta move in which you say, oh, there are certain machines where the rules themselves can be written as data on the tape along with the inputs. And that means that those universal machines can perform any computation, not just the computation that their table of rules tells them to do. So, you know, what I've just described is exactly what a computer is. A computer is a universal Turing machine. And when we say that it follows an algorithm or runs code, the code is exactly that written out table of rules, you know, that is actually written in the memory and that is read by the machine. So it's a really interesting move, and it's one that allows for arbitrary behavior to be done by a system that is itself quite simple and has fixed complexity.

Now, if we just think Turing machines generally, meaning not universal Turing machines, but just any old Turing machine, all that is is something that is abstract, that has a state, that can take inputs, change its state, and produce outputs. Now, that is such a general construction that it applies to anything in nature that you want to think of that way. It could apply to Jupiter. It could to the water in a stream if we define state input and output in the appropriate way. But a universal machine, in other words, one where the directions that it's following define the things that it does, that's something a lot more particular. And that really begins with life.

I should also say that there is a key concept here. It's a little bit deep mathematically, but it has to do with the mapping from microstates to macrostates. The reason that time has an arrow, that the past is the past and the future is the future, has everything to do with entropy, with the fact that things go from more ordered to less ordered states. So if, for instance, you're looking at a bunch of balls bouncing around on a pool table, but suddenly you see them all come together into a triangle and one ball shoot away with all of the momentum, now you know you're watching the movie in reverse.

And it turns out that computation and entropy are also very closely related and that you can think about computation as really being the story of how causality comes into the world. And when you look at any computer language that's ever been invented or discovered, they always have something like an if-then in them. And the reason that they have an if-then is because they need to be able to create causes in the world. Anything that is a behavior, that is a computation, requires conditionality. that you do something on the basis of what was input.

Even if you think about a thermostat, when the temperature drops below a certain level, it will turn on the heat. And of course, the heat will then cause the temperature to rise, which will then cause the switch to turn off. So that's an if-then. You know, it's a minimal form of behavior. The behavior of a thermostat is conditional on the temperature. So, you know, that's sort of the elementary building block behind anything that can be said to behave. And it's also an elementary building block in anything that can be said to compute. The science of causality is a way to think about computation and how computation arose in nature, and especially in the emergence of life, as opposed to just being about computer technology that we began making in 1945.

Now, there's another really interesting chapter to the story of computing itself, which was really invented by John von Neumann. Now, John von Neumann was trying to work out the theory of how it is that a thing can reproduce. He was trying to solve the Baron von Munchausen problem, you know, pulling yourself up by your own bootstraps. How could life bootstrap itself when you need to have an instruction tape in life and that instruction tape needs to have the instructions for building a self? But, you know, if the self is copying the instruction tape, then, you know, it would seem that you have to have an infinite regress. You know, in order for you to be able to construct yourself, you need to kind of have a copy of a copy of a copy of a copy of yourself. And this is kind of like a logical paradox.

So the way von Neumann solved this was by saying, well, if your inner instruction tape has the instructions for making two different machines, as he called them, machine A and machine B, machine A would follow the instructions on the tape in order to build whatever it says, and machine B makes a copy of the tape. In other words, it treats those instructions as symbols rather than treating them as instructions. And if the instructions for building both machine A and machine B are themselves also on the tape, then you kind of break the paradox. You have a system that can reproduce itself.

This is really cool because he figured it out in the late 1940s. And this was before Watson, Crick, and uncredited colleagues had figured out the structure and function of DNA, which is exactly this instruction tape that he was talking about. But also before the discovery of the ribosome, which is what he called machine A, or the discovery of DNA polymerase, which is what he called machine B, both of which indeed have the instructions for building them in our DNA. So he got all of this exactly right. And also, he realized that this was basically the recipe for a universal Turing machine. In other words, a computer. Machine A literally is following the directions on a tape in order to perform certain actions. And that's basically how Turing defined computation.

"The birth of biology is the origin of the first computers."

So what this means is that the original symbiosis that gives rise to an organism that can reproduce itself heritably is literally the coming together of a computer. So in that sense, biology is computer science. The birth of biology is the origin of the first computers. And what we have today in our phones, our laptops, and so on, we should call artificial computers in the same way that we call AI systems artificial intelligence.

We know from von Neumann that life has to be self-constructing. In other words, it's got to be able to grow, heal, and or reproduce. Otherwise, it will no longer exist in the future. If it's complex, if it's more than just a really trivial crystal, then that requires general computation in order to build itself. And in particular, you need the heritability property. In other words, you need it to be the case that it's got a set of instructions inside itself such that if those instructions are changed, then future generations will also change how they are. Right. So there's got to be this kind of persistent instructions that allow heritability to occur. So life is general computation, or I called it playfully computronium, meaning matter that computes.

And the other part of that definition is that, you know, once you've got that property, if what it's interacting with is other life, which will always be the case, by the way, you know, because the moment life can reproduce, suddenly it's in a bath of other life, then that opens the possibility for something called symbiosis.

Darwin was always troubled by the question of how life gets started. We tend to assume that life evolves through random mutation and selection for fitness. So this is called the neo-Darwinian synthesis. It's not exactly what Darwin came up with, but it's what I guess Darwin settled into as he was interpreted by later thinkers. So the idea here is that we have a genome, and that genome gets hit with radiation, randomly mutated, and those random changes stick to the degree that those random changes confer some survival advantage. Now, that story is not exactly wrong, but it's only half the story.

As it turns out, the big steps in the advancement of life have come from cooperation, not competition. They've come from entities, two or more entities, sometimes many entities, with either the same genome or different genomes coming together in an act called symbiosis. Symbiogenesis is when two or more life forms come together to make a more complex life form that itself can reproduce. If it cooperates with the life around it, then it can become obligate, meaning that those cooperating things have to now reproduce together.

Now, the story of symbiosis, which is to say that things come together, cooperate to make bigger, more complex things, turns out not only to address the problem of how life becomes becomes more complex over time, but also the problem of how life gets started. So it turns out that when molecules cooperate, meaning molecules somehow end up in relationships with each other, chemical reactions, whereby those reactions lead to more of those molecules, and then those cycles of chemical reactions, which are autocatalytic, would be the technical term, start to interact. Those can make systems that in turn propagate themselves. That story of symbiosis going all the way down to molecules is not only the story of how life becomes more complex, but also of how life arises in the first place.

And this story of cooperation is not what we generally think of when we think of evolution. We think only of competition. But cooperation is how we went from molecules to bacteria. It's how we go from bacteria to eukaryotes, the kind of cells that we're made out of. It's how we go from eukaryotes to multicellular organisms like us. And it's also how we've gone from individual humans to societies. And that cooperative story is just as important as competition. In fact, in some sense, cooperation and competition are not even distinct. They're parts of the same larger story.

Lynn Margulis and the major evolutionary transitions

The idea of symbiosis dates back at least to the beginning of the 20th century. But the person who really brought it into the mainstream and proved that it was true, that it really had happened, was Lynn Margulis, who was a real kind of rebel biologist. The paper in which she described this, it's from the late 60s, was rejected by her account from more than a dozen journals before it was finally accepted in the Journal of Theoretical Biology. And it was very controversial for many, many She eventually won the National Medal of Science in 1999, but it remained a very controversial story right through most of the 20th century. By now, it's accepted by pretty much all biologists.

And what Margulis and others before her had noticed is that mitochondria look a lot like certain kinds of bacteria that we can find free swimming in the world. They also, as it turns out, have their own DNA. So there are little loops of DNA inside the mitochondria, and those loops look kind of like bacterial DNA. And what Margulis showed was that, you know, indeed, there was a common lineage between mitochondria and free-swimming bacteria that are out in the world. But she believed that symbiosis, this fusion of simpler life forms into more complex ones, was not just something that had happened with mitochondria, where she proved it, but was the engine behind. All biological development, all biological complexification. And that is still not mainstream among a lot of biologists.

However, in the 1990s, there were two, I would say, very prominent, more mainstream biologists, Eörs Szathmáry and John Maynard Smith, who at least formulated this idea that there had been some major events in evolution, including the formation of cells, including the formation of societies and hives of insects and so on. They called them major evolutionary transitions that were clearly symbiogenetic. In Szathmáry and Smith's original paper on the major evolutionary transitions, they listed eight major transitions. So, you know, we go from Lynn Margulis to John Maynard Smith and Eörs Szathmáry.

And my perspective is more like Lynn Margulis's, that those big events are extremely significant in the history of our planet, but that symbiosis is actually happening all the time and is the engine that is kind of ongoing and that is building life. Life has a selection process that is more like classical Darwinian evolution, but it also has a revolutionary process where things are changing in ways that involve recombinations of things that are more active. So this is where I say complexifies through symbiosis. So life is matter that means. It's matter that has folded itself into a computational structure that can replicate itself, that can construct itself, and that in turn cooperate with other matter that has done the same thing in order to make more complex life.

And we see evidence for those kinds of symbiogenetic events, you know, which are not the big ones that Smith and Szathmáry identified in stuff like DNA. And I'll use human DNA as my example here. So, you know, we first sequenced a full human genome back in 2001. And one of the big surprises in the human genome. is that only about 1.5% of it codes for the proteins that make up our own bodies. So what is the rest of it? This is often called junk DNA. It's not really junk. A lot of it is involved in regulation and in other functions that we don't actually fully understand.

But when you start to explore all of that sort of 98% of the remaining DNA, what you find is that a whole lot of that so-called junk DNA is actually the remnants of retroviruses that appear to have injected pieces of their genome into our genome and moreover into our germline, the part that reproduces from generation to generation. And so the HIV virus is an example of one of those retroviruses. Now, what HIV doesn't do is to insert itself into the DNA of eggs and sperm. But if it did do that, then not only would it infect you, it would infect all of your offspring as well. Which is kind of a creepy thought. But when you look at what our DNA is made out of, most of it appears to have been made out of DNA that had exactly that historical evolutionary trajectory, that it was injected in by other things. So in other words, our own DNA is this kind of overlay or palimpsest of the DNA of all kinds of other entities that reproduce inside DNA that have, in some cases, jumped between species. and so on.

And actually, we have a lot of evidence that those pieces of DNA, which might originally have been antagonistic or might have been infectious, have taken on critical for us. So for instance, there is a virus called ARC, A-R-C, which is endemic to all mammals. And it turns out if you knock it out in mice, those mice stop being able to form new memories. So somehow this piece of viral DNA, which still forms virus-like structures in our own bodies, is intimately involved in memory formation in our brains, which is kind of wild.

Or there's another protein that fuses together cell membranes. It's a little bit like the proteins in RSV, which is a respiratory infection that infants and some older folks get as well. And that fusing together of membranes turns out to be what forms the placenta. The placental barrier is formed by fusing together membranes of adjacent cells to make a kind of barrier. And that also looks like it was the product of some kind of retroviral invasion a long time ago.

So, you know, when you start to look at those sorts of events, actually life stops looking like a tree at all. It starts looking more like a tangled bush that is always kind of fusing and coming back together with itself as well as diverging. And so as we start to understand more about how evolution works, that these cooperative symbioses are really important, that it doesn't work like a tree, but more like a bush that intersects with itself, that fuses all the time, things come together all the time, it changes all sorts of things of things.

So one assumption is that organisms are passive and mutation is some random phenomenon. That seems not to be the case, you know, since a whole lot of the changes are active. They involve things, you know, literally injecting themselves into other things or the sharing of code, you know, back and forth. It's more like, you know, like an open source code repository like GitHub, where people are constantly, you know, importing libraries and sharing code with each other. You know, the code of life works like that too.

And that actually implies a lot more agency on the part of life itself as well. There's a great historian of science, Jessica Riskin at Stanford, who has written about this. She actually just wrote a new biography of Lamarck, who we all learned was wrong in high school because he believed that giraffes have tall necks because they stretch their necks up over a revolutionary time to reach the high parts of trees and they self-constructed. And of course, we know that that literal form of self-construction is not true. The length of a giraffe neck is determined by its DNA. But the deeper point that Lamarck was making is that he believed that life constructs itself, that life has a lot of agency in evolution.

And the contrary view was actually a religious view that living stuff has no agency because all of that agency belongs to God. And so in that sense, at least everything that is non-human doesn't have any kind of will. It's purely acted on by external forces. Evolution just selects for whatever is fittest. And that's not the case, right? If you look at genes infecting each other, at genomes that are combinations of other genomes, at things that actively edit themselves, it's a much more active story. And it's a story that also includes a lot of elements of cooperation as opposed to competition.

Another thing that I think comes to the fore is that when you have entities that cooperate to form a larger entity, when you have a symbiogenetic event, suddenly the game that is being played changes. The roster of players changes and new evolutionary niches open up, new possibilities for play, if you like, open up. And what that means is that this idea of fitness, of there being a fixed fitness landscape and everything optimizing on that fitness landscape is also wrong. Because it turns out that the landscape moves along with the players. You know, all of the players are each other's landscape, if you like, and the players are constantly changing because of their shifting cooperations and fusions and mergers.

So all of these assumptions about Darwinism, that there is such a thing as fitness in any fixed sense, that you can rank two entities based on which is fitter than the other. You can't because it depends on their environment and on the context and on how they're cooperating or competing with each other at that moment. The idea that organisms have no agency and are just acted on from without and that evolution doesn't have a guiding hand, if you like. No, it has many guiding hands. It's those organisms themselves that supply the guiding hands. All of those assumptions are wrong. And I think that they've infected many other fields as well, everything from game theory to economics. There's, of course, social Darwinism, which we know leads to all kinds of pretty ugly conclusions. All of those are not only morally wrong, they're actually wrong from a scientific perspective as well, because they're based on a mistaken idea about what evolution actually is and how it works.

So, yeah, life is self-constructing computation that complexifies through symbiosis. Now, I'll bring this to its logical spicy place, which is, is AI alive? I've mentioned the two, in my opinion, big founding fathers of computer science, Alan Turing and John von Neumann. There is a difference in the way they thought about computation that I think is important. So for Turing, the Turing machine is an abstract thing. It's an abstract head that moves left and right along a tape and reads and writes symbols on that tape. But it doesn't actually have the ability to read and write. A copy of itself. It's not self-modifying. Whereas for von Neumann, this universal constructor is something that can literally build another universal constructor. In other words, what it's reading and writing is atoms, not just symbols.

And that's important because, you know, if you think about a phone, for instance, it's a universal Turing machine in the sense that it can compute anything, but it can't pop another baby phone out the bottom, you know, the way half of us. can. So that ability to literally make a copy of yourself or do what all of us can do, which is to grow and heal, that's also a form of self-construction. That's something that these universal constructor-based von Neumann machines can do, but the Turing machine can't do. And that's important because, you know, life is not just abstract computation that involves, you know, symbols in some abstract mathematical space. It's, you know, literally the reading and writing of the things that you are made out of into the world.

But I should also point out that, you know, if we look around at the number of phones in the world, there are clearly more phones this year than there were last year. So, you know, where are they coming from? There's a kind of silly quote from an astrophysicist, Robert Jastrow, in the early 80s. He said that we are the reproductive organs of computers. I think, you know, in some sense, that's correct. You know, AI, in a similar sense, doesn't exactly create more AI in isolation today. So it doesn't have that, you know, that basic self-constructing property. It's all made on Turing machines, not on von Neumann machines. The things that computer programs, including the ones that run AI models, read and write are digital symbols, not actual. computers, right? Not the wires and silicon that make up the actual physical basis of computers.

However, we are certainly making a lot of computers nowadays. There's a lot of data centers getting built. So, you know, when we look at the system more broadly, you know, you can think about AI as being extra brain tissue that is being made by a larger living system, which is, you know, us and our entire technosphere. And in that sense, steam engines, computers, AI, they're all part of the same big story. You know, this idea that we are somehow separate from nature, that everything that's been going on in the last few hundred years is unnatural, I think is another of those fallacies that comes from misunderstanding how nature works in the first place. This is actually part of the same story.

It's kind of a matter of which lens or which perspective you look at it through. You know, in the same sense that you can look at an insect colony and you can see a superorganism made out of insects, or you can see a bunch of insects. You know, in the same way, you can look at a eukaryotic cell and you can see either one entity or many entities, all of these, you know, bacteria on the inside, which are the mitochondria, cooperating with the bacterium on the outside, the big cellular membrane that they're all inside of. And both of those perspectives are true.

So, you know, symbiosis in a way is about naming the assemblies that things coalesce into as new things onto themselves that are more complex in terms of how they function because of a sort of division of labor of all of the parts. That's sort of a general recipe for symbiosis. And when you look at it that way, the story of human society over the last 10,000 years just looks like another of those symbiogenetic events. You know, yes, we're all still individuals, but also we are something bigger than that when we come together and we're capable of so much more when we come together in just the same way that bacteria coming together into a eukaryotic cell are capable of something much more complex than individual bacteria are.

So the thing that characterizes living systems as opposed to non-living ones is function. Functionalism is the idea that things are what they do for other things, if you like, that everything is about relationships as opposed to being about some inner essence. If you think, for instance, about a rock. If you break a rock in half, we don't say, oh, the rock is broken now. It doesn't work. A rock broken in half is now two rocks. Whereas if you break a kidney in half, you now no longer have a kidney. It no longer will function. And that's a property that is true of life, of living systems. They have purpose, they have function, as opposed to just being matter.

And what does that mean? What it means to have purpose or function is basically to be in symbiosis with other things that also have purpose or function and whose inputs and outputs somehow are hooked up to yours. So, you know, your outputs might be their inputs and vice versa.

And Alan Turing, who was one of the founders of computer science, he was a mathematician. So he was thinking about mathematical functions. If the function of some system is, say, to add numbers together, then his point was that you could make a machine, a physical machine that would do that out of cogs, out of electrons that move around on wires, out of people waving flags. It didn't matter as long as the right transformations were happening in the information in that system. So that idea that it's the information that matters and the substrate can vary is something that he called substrate independence.

And this is really important because, you know, if you have something like a kidney, when we think about what a kidney is, it's really functional. Its job is to take the urea out of blood so that the rest of your body survives because too much urea accumulates in the blood. It's toxic. So we're able to build something like a dialysis machine in order to substitute for a kidney precisely because it's the function of the kidney that matters, not what it's made out of. Now, obviously, you know, a dialysis machine isn't as good as a kidney in a variety of ways. You know, nobody who has the choice will choose a dialysis machine instead of a kidney. But the fact that it keeps you alive is kind of passing the kidney Turing test, if you want to think about it that way. It's saying that when you substitute something that has the same function, then it works the same way as the original thing.

And nature has played this trick many, many times. You know, when you look at how different functions are carried out in biological systems, you see that nature is a great inventor. Flying, for instance, has been invented a number of times in nature, all of which are made very differently, but they all serve the same function. They all fly. So, in other words, there is something profoundly ecological about the idea of function and profoundly symbiotic or symbiogenetic that function is defined relative to other functions and functions work insofar as they work with others.

So, for instance, I kind of need to wear glasses, although I usually don't nowadays. But the fact that the lens is made out of glass as opposed to out of the transparent cellular material that our own biological lenses are made out of is not the important thing. The important thing is that they refract the light in a certain way. And seeing is all about that light getting refracted the right way. It doesn't matter what it's made out of. If we substituted the glass for quartz of some other kind or for plastic or what have you, it wouldn't matter. These might have different properties in terms of their weight, in terms of how well that ecology works on the tops of your ears or your nose, but it doesn't matter as far as vision is concerned.

So this idea of substitutability, of one substrate or one pathway being able to take the place of a different one is something that nature uses all the time. as well. When you look, for instance, at metabolism inside a cell, the point is to make a molecule called ATP that carries energy, and we have multiple chemical pathways in cells for doing that. So, you know, In some sense, biology is the origin of function. It's kind of an ecology of functions that work together to make bigger functions. And the platform independence, in other words, the fact that you can substitute different materials, different processes for how those different functions work, is fundamental to how life works. Because it's all about the relationships, not about how it's done or what's inside.

And I'll bring this to its logical spicy place, which is that a lot of people will, you know, are still saying things like, you know, AI is just running on a computer, like a computer can't be intelligent. That's a property of brains and, you know, living systems. And. And I guess, you know, the reason that that doesn't strike me as correct is because, you know, the fact that a system is living is precisely the fact that it is functional, that it serves these ecological relations with other things. You know, when neurons are talking with each other with their electrical signals as they do and chemical signals, they absolutely are substitutable for other kinds of entities that are doing that same kind of interaction with each other.

There's a thought experiment. It's been articulated by a couple of philosophers, David Chalmers, Susan Schneider. They call it the brain of Theseus by analogy with the ship of Theseus. The ship of Theseus story being if there's a ship and you start to replace its timbers, does it become not the same ship at some point when all the timbers are now different? And the answer, I think, is no. By any reasonable definition, if you replace the timbers continuously, it's still the same ship. That's certainly the case with us. All of our molecules are replaced as we go through life. It's still us. So we are literally the ship of Theseus.

But their version of this is a brain of Theseus, where you imagine taking a neuron. And replacing it with a computer that has the same inputs and outputs. And, you know, we kind of know in theory how one might do that. Brain-computer interfaces are very difficult to make, but we sort of know how individual neurons compute. That's been sort of worked out since the 1950s. And so, you know, the question is, if you replaced one neuron in your brain with a computer wired to the inputs and outputs, Would you feel any different? And, you know, I think the answer is pretty obviously no. But then, well, what if it's two neurons? What if it's 100 neurons? What if it's a million neurons? What if it's all of them? You know, would there be some moment at which, you know, you would suddenly, your consciousness would fade out or you would stop, you know, being truly intelligent or whatever? And for me, I think the answer is very clearly no.

"Because the you is in that ecology, in those relationships, not in the particular molecules, not in the particular cells."

You know, if you did that perfectly, if you maintained all of those relationships, all of those input-output relationships, then nothing would feel any different. Because the you is in that ecology, in those relationships, not in the particular molecules, not in the particular cells. It's the organization of them, and it's not some magical property of those molecules. of the matter. So, you know, that's functionalism in a nutshell.

And to me, you know, if you sort of pair that together with Turing's big insight about, you know, this being essentially computation, you know, behavior is something that can be expressed in terms of computation, there's not any very profound difference between, you know, between computation happening on a silicon substrate and computation happening in a neural substrate. There are certainly differences in terms of the efficiency. Biology is still in some ways much more efficient than silicon. It's actually one of the things that my team is working on today. And also, by the way, reliability, which kind of goes along with efficiency. There's a precision about the way we compute in silicon that's really a function of the fact that we use the same kinds of chips to calculate spreadsheets and stuff. And we don't want anything to be off by a little bit when we do a sum in a spreadsheet. It's probably an error to be bringing that same kind of precision to neural computation, by the way.

But if we kind of... Defocus our vision a little bit and forget about those details. And we just think about that ecology of relationships. Whether you do it in silicon, whether you do it in proteins, doesn't matter. For example, in the movie, The Hail Mary, there's an encounter with an alien who is made out of kind of mineral or rock. And, you know, Of course, the substrate is totally different, right? Rocky doesn't have neurons in the same way that we do, isn't made out of proteins in the same way that we do. But, you know, most of us would not say, well, therefore, Rocky can't be having real thoughts. You know, therefore, He can't be conscious. Of course not. And in the same way, I think that's true of computer systems, of any other substrate. It could have been done with Babbage's steampunk computer that relied on cogs and wheels and belts. And the whole point is that when you look at things from a functional standpoint, it really doesn't make any difference.

So the thing that life has that non-life does not is function or purpose. And in particular, the parts of a living thing have purposes that they serve for each other. You know, the heart is there to pump blood. The lungs are there to breathe air and to, you know, put. Oxygen into the blood and get the carbon dioxide out. Kidneys are there to get the urea out and so on. So, you know, all of those things, functions or purposes, are what make life, life.

And this is also something that, you know, that a lot of scientists have struggled with because there was this idea in the 19th century that we should get rid of any idea about teleology or purpose in nature. That, you know, nothing has any purpose in nature. It's just atoms and molecules. It's physics. It's chemistry. You know, this idea that purpose doesn't exist in nature was kind of seen as a way of trying to remove religion. From nature, you know, or remove Aristotle's idea that air wants to rise or something like this. But it actually, I think, had a really perverse consequence. It's kind of like trying to get the lump out of a carpet by flattening it. And what you end up doing is making a really big lump in one spot because you've tried to iron the rest of it flat.

When you try to get rid of purpose, what you end up doing is essentially moving all of the purpose into one spot. And, you know, in the case of the 19th century biologists, that one spot was God. Or human purposes, human will, which was seen as a reflection of God's will, a little piece of the divine, that we all have souls and that's what gives us purpose. No scientist, of course, would have accepted that, but we kind of moved this idea of purposes outside the purview of science and thereby either we're pretending that purposes don't exist or that they come from outside science and are supernatural. When, of course, the reality is that whenever you have living systems, they're always serving functions for each other.

And in real life, of course, if you're a biologist, you can't talk about something like the immune system without talking about its function, about what its purpose is, what it does. It's there to defend you against bacteria and viruses that would otherwise kill you. You can't talk about the function of the hairs in your nose without... without talking about why they're there, what they're doing for us, right? They're there for filtering out certain crud before it can get into our lungs, and so on. So this idea of functionalism and of the fact that functions can be served in various different ways by different kinds of mechanisms, That's the very essence of life. And life is the emergence not only of causality, of one thing causing another, but also of purpose in that causality.

I should mention here that we've actually done some really cool work on my team in the last year or so in artificial life, which artificial life, by analogy with artificial intelligence, is setting up. Sort of proto-biological systems in a computer. And this experiment is one that really investigates abiogenesis, which is to say the emergence of life out of non-life.

Here's how it works. We begin with very, very short random programs. So they're only four instructions long, four bytes long, and those four bytes are random. The idea of a random program might seem quite paradoxical, but if you look at what a program is made out of, it's just made out of instructions, one after another. And in this very simple language, those instructions are exactly one byte. So, you know, when I say they start off as random programs, all I mean is random bytes. that do something. And where do I get off with the idea that they do something? Well, molecules do something as well when they interact. If you have two molecules that interact and one of them Phosphorylates the other, that's doing something, you know, or transfer, you know, transfers an ion or, you know, breaks off a hydrogen or whatever it is, right? So all chemical reactions are, you know, one molecule doing something to another.

And so that's all it is. And the reason that I used a minimal programming language for this was just to ensure that the set of instructions that I was working with were sufficient to, in principle, allow any kind of program to work. In this case, I based the first version of this on a computer language called Brainfuck. It wasn't named by me. This was named by a Swiss grad student in the 90s who invented it. So his idea was to invent a computer language that was really minimal and that had a minimal number of instructions. And he called it that because... It is a bit of a brain fuck to try and program, and it's very difficult to program in. But anyway, the reason that I used it was because of its minimalism.

And so you start off with very short, random tapes that don't do anything. But this particular variation of the language is one where the instructions can self-modify. In other words, they can modify the tapes themselves. But in the beginning, they don't do anything interesting because they're just random instructions, right? It's like a chemical soup of just random molecules bumping around together.

The procedure is as follows. You have a soup of, let's say, a thousand of these little tiny random tapes. You take two of them out of the soup at random, and you stick them end-to-end, and you run them. So this is sort of an interaction between two molecules bumping around in that soup. And then once you've run them, you pull them back apart, and you put them back in the soup, and then you just do it again. You mix and you do it again. And you just do that over and over. This doesn't seem like a very promising way to make anything interesting happen. You know, take a bunch of really short random programs and, you know, that have the potential to read and write to themselves, but, you know, they're literally not doing anything of any interest in the beginning and just do it over and over. What do you expect to happen? Nothing.

But what does happen is actually really remarkable. So after a few million interactions, suddenly you begin to see these tapes becoming longer and more complex, and they start to really compute. A lot of computation starts to happen, and they start to reproduce. Why? Well, if you think about it, a program that replicates is going to out-compete a program that doesn't replicate. If you have a program that can replicate, it will exist in the future. Whereas if you have a bit of code that can't replicate, it will eventually get overwritten by a program that can. So in some very trivial way, evolution is already at work with respect to even non-functional programs in the beginning because anything that can replicate itself is going to persist into the future, whereas something that can't won't. So, you know, It's a sort of model for how life arose, and it shows you that life can arise actually very easily in this minimal form.

And I should also point out that the fact that these tapes began very short, but that you stick them together in order to run them is really important to this experiment, because what that allows to happen is symbiosis. It allows separate tapes to come together to make a longer tape. And that's essential in this case because the shortest program that can reproduce is actually longer than four bytes. It's more like six bytes or so. So the fact that you're putting tapes together is necessary in order to get things that are long enough and complex enough they can reproduce. And once they're reproducing, you find that those in turn... Come together to make even longer and more complex states that themselves can reproduce. So you can actually see symbiosis leading to life and then complexifying that life, even in this very, very simple artificial system.

And when you look at a BFF soup, after it's been running for a few million interactions, you find programs in it that are definitely purposive, that look like they were made by an engineer. I mean, they look like they were made by a mad engineer because they're not programmed in the way that you rationally would as a programmer, but they certainly are doing something purposeful. And if you break them, if you change some byte at random, you will generally destroy their ability to function and they won't be able to replicate. And that modified one, that destroyed one will get overwritten by something else.

So it's really what is so magical about seeing this process occur is that you start off without any You start off with just bytes, and then after a while, through nothing but random interactions, there's no externally imposed fitness function, there's no judge, there's no designer, there's no programmer. But after a few million interactions, you see programs that are purposeful, that look as if they were engineered. And all of that has just happened through interactions between things that, you know, when they're lucky, they serve a function for each other, if that makes sense. Right. So when, you know, when a couple of instructions come together that can help each other to continue to exist in the future. Then. They will continue to exist in the future. And that's the key, that's the secret behind symbiosis.

And as those pieces become more and more complex, of course, they also become more and more functional. What they do is more and more sophisticated and more and more contingent on everything else that is in the soup as well. So, you know, you can really see how not only does, you know, can life truly emerge from nothing, which is the thing that had really bothered Darwin, but you could also see how and why it becomes more complex over time.

So a simple BFF program just replicates itself. It doesn't have any more complex behaviors. But we've run some experiments in which we require those programs to, you know, do a little more work to earn their keep. So, for instance, you know, it might be the case that they need to chase a light, which is actually giving them the energy to do computation in the first place. And by the way, you know, I'm not sure I've mentioned this, but computation requires energy precisely because it is an entropy lowering process. It sort of erases information from the world. And in the process of doing that, you need to feed it free energy. That's why plants need to photosynthesize. It's why we need to eat high energy foods like fats and sugars.

And so it has to have behaviors that are about more than just copying its own matter. And those behaviors might involve chasing certain chemicals, fleeing negative conditions that are going to destroy it, like heat too high or something along those lines. So all of that requires that the computation become more complex, more sophisticated. Also, of course, one of the great sources of energy, if these little guys need energy to compute and they can store it, is other little guys. So you've got predator-prey relationships. Those kind of emerge naturally in any system that has life. So the moment you've got stuff that wants to eat other stuff, then the stuff that is potentially going to get eaten needs to try and escape as well.

So both because... because of cooperating with other life and because of competing with or even eating other life, life needs to start to model its environment and model other life. And by model, what I mean is be able to behave in ways that are going to have positive effects for your own future, given what you expect the environment and others will do, conditional on what you do. So this kind of sets up a sort of arms race of intelligence in which the more I'm able to model, the more I'm able to predict what others will do under various circumstances, depending on what I do, the better I will be able to survive.

I should mention here that we've actually done some really cool work on my team in the last year or so that shows that in a multi-agent system, so this is machine learning work, in a multi-agent system where every entity is able to model itself and model others. As part of its model of the environment, those systems are able to learn how to cooperate in ways that they would not be able to if they weren't doing this kind of self-modeling and modeling of others. So you've actually got to be able to not only think conditionally about how the environment will react to things that you do, but also how you will react to those changes. And if there are others in the environment that are like you, You've got to be able to predict their behaviors, keeping in mind that they are going to be like you as well, that they're similar to you. So all of that comes together to basically create evolutionary pressure for entities that are smarter and smarter because, you know, your ability to model others is going to be increased by the complexity of your model and your ability to learn. But the more you learn, the harder you will be to model as well. So there's a kind of, you know, friendly and unfriendly both arms race of intelligence that arises naturally whenever you get living systems.

When we were kids, we all knew what artificial intelligence meant. There was Rosie, the housemaid robot in the Jetsons, and there was HAL 9000, and the Star Trek computer, and all kinds of robots or AI systems, embodied or disembodied, that were intelligent. And that was what AI was. Now, sometime in the 90s, the term artificial intelligence started to get used for systems that were very narrow in what they could do. Things like digit recognizers. They could look at a bunch of pixels and distinguish zero through nine or distinguish between a fixed set of visual objects. Is this an apple, a pear, a Siamese cat, whatever.

And that was, I think a lot of people felt like that was an abuse of the term artificial intelligence. Certainly it required that we come up with some way of distinguishing digit recognizers from Rosie the robot. And that was why we invented this distinction between artificial narrow intelligence, meaning a system that was designed for doing a specific task, and artificial. artificial general intelligence, meaning when you could just have a conversation with about anything. So that was the original purpose of the split. I should also say that the reason that artificial narrow intelligence was called artificial intelligence at all is because a lot of it was neural network based. There had been many attempts to do things like distinguish digits using handwritten programs, and those attempts didn't work. So they were neural in some way. They were brain-like in some way, but they were very specific in what they could do.

Around 2020, a little bit before, we started to have language systems, so large language models, that were based on pre-training with a very large amount of text from the internet. And it turned out that you could start to have conversations with those systems about... Arbitrary stuff. You could ask them to write poetry or critique some prose or think with you about something, tell jokes, whatever it is. And they weren't specifically trained to do any of that stuff. They were just trained to model language. In my mind, as soon as we started to get systems of that sort, they were no longer narrow intelligence.

You know, I remember seeing John Oliver saying, you know, don't worry, these systems are still just generating text. That's not general intelligence. It's just text. To me, that doesn't make any sense. whatsoever. The whole point of text is that it is general. You can read and write about anything. This whole conversation that you and I are having, most of it could be captured by the transcript, by just text. So text is not narrow. Text is general.

And in my opinion, if you took any of today's AI systems based on large language models and you transported them back in time to the year 2000, when that distinction between artificial narrow intelligence and artificial general intelligence was first being made, I think anybody back then would have said, yeah, you're there. You've got it. This is what we meant by artificial general intelligence, by AGI. I wrote a piece together with a colleague, Peter Norvig, where we said artificial general intelligence is already here. And I certainly felt that that was easily true in 2022, even more true now. And I think we crossed that threshold when we stopped training those systems to do specific jobs and started training them on language.

So, you know, to be honest, I've actually been really confused by why it is that we are still, you know, waiting for Godot. You know, like, why are we still saying, you know, artificial general intelligence is out there in the future? What do we even mean by that? It seems like we've quietly shifted the goalposts. And maybe what we mean now is artificial superintelligence, meaning, you know, systems that are as good or better than humans at every possible kind of task.

It's true that sometimes LLMs do weird stuff. You know, they will hallucinate something strange, you know, or they'll fail to understand something, you know, that seems pretty basic, right? Or to follow an instruction correctly or something. But if we were having those interactions with Rocky Rocky the Alien and Hail Mary, you know, the conclusion would not be, oh, well, clearly Rocky is not actually intelligent. You know, it's more like, oh, you know, there's some funny gaps, you know, both superpowers and gaps here that are a little different from ours. And that should be no surprise. I mean, the... Both the way our brains are made and our developmental trajectory and the way our culture works and the way we're brought up, it's all really, really different from the way we pre-train and post-train AI models. So it would be really surprising if everything were the same, if all of the strengths and weaknesses were the same.

But it's part of maybe our nature as humans to have this sort of... anthropocentrism. And we think about our strengths and weaknesses as being the gold standard. This is what it means to be intelligent, to be whole, to be what we are. And any deviation from that must mean that it's not the real thing or something. That's, of course, something that falls apart pretty easily when you look at it more closely. I mean, for one thing, there's a huge variety, actually, of human intelligence. I mean, you know, it's not like all of us have the same abilities and capabilities either. And we also, you know, make a lot of mistakes and give each other a lot of grace for those mistakes in ways that we absolutely don't give grace to an LLM. It's like we're waiting for a guy. gotcha, you know, whenever they get something wrong. But whenever we get something wrong as humans, it's sort of like, oh, well, it's just, you know, like, maybe you just didn't have coffee or you misheard me or, you know, we overlook it.

I had a debate about this on stage with a good friend, Melanie Mitchell, and she talked about AI models not being able to get the right answers to certain kinds of simple logic puzzles. And I knew she was going to do this. So I had actually made up a couple of those logic puzzles, and I gave her one on stage, and she got it wrong. And, you know, this doesn't mean that Melanie isn't intelligent, right? And Melanie is brilliant. But, you know, of course, we all get things wrong now and then for whatever reason.

Or, you know, maybe another example would be useful. Our team was one of the first to test LLMs on medical diagnosis. And when we first did this, we found that the models would diagnose things correctly only, I don't know, half the time or something like this. And the conclusion from the team at that point was, well, I guess this isn't ready for prime time. These things are not ready to be doctors. But, you know, when we went and did the human tests, the human trials of like, well, how often do doctors get diagnoses right on the basis of, say, a description, you know, of all of the symptoms and so on? It turns out that, you know, doctors get things wrong a lot. as well. And actually, the models were already doing slightly better than doctors in general. So, you know, that doesn't mean don't see doctors. It doesn't mean, you know, it does mean maybe, you know, if it's important, you might want to get a second opinion.

But, you know, intelligence doesn't mean getting everything perfect, being able to do everything right, etc. It means that you are able to solve a variety of different kinds of problems a lot better than you would be able to if you didn't have some underlying model of how the thing works that you're talking about. If it's multiplying two numbers together, it's not just that you've memorized the multiplication table, but that you understand what multiplication means, and you have something like a multiplication algorithm in your head that you can apply to a new problem. When we find that AI is able to start advancing the frontier of mathematics, for instance, and there have been some recent results in just the past few months that show that, right? Novel theorems, novel proofs. I think that's a really nice example of cases where you can't say this is just regurgitation, that it's just sort of recalling something because they've been trained with a massive amount of training data unlike humans. The very stuff that we think of as intelligence, right? Taking what we already know, taking what we already have been exposed to, putting it together in a new way to do a new thing.

So I think that restricting ourselves to thinking about AI as tools is probably not quite right. You know, a tool is something that That's no agency or that we model as not having any agency and that doesn't do any thinking. But of course, you know, AI is exactly not that, right? It's something that also thinks as the human brain thinks. It's something that also participates in that big sort of human ecology.

So when we increase the amount of intelligence in a system, we greatly increase what it can do, what its possibilities are. You know, the fact that. That human life became so much better over the period from 1850 to 2000 had a lot to do with rising human populations. There's just a lot more brains working together. And not just a lot more brains, but a lot more brains freed to do real thinking. When most people were still doing subsistence agriculture, the huge majority of their efforts were expended just on getting enough food to eat. And it was only really a small percentage of elites who could think about science and technology and art and culture and so on. And now there are so many more of us who can do those things thanks to that division of labor and thanks to all of that unleashing of new sources of energy and new resources.

So, um, You know, my wish, you know, out of this whole AI revolution is for that party to continue. And that should be all of our wishes. I mean, that's what progress is. And again, I don't think that progress is just some new invention that the economists have informed us about from 1800 onward. It's that whole evolutionary story. There is a real progress there where complexity rises, makes systems that are more and more intricate, more and more intelligent, and that have more and more degrees of freedom, that can do more and more.

I do think that part of the problem is that we have this idea of essentialism, you know, that we are what we're made out of, or there's something about us physically, embodiment, you know, etc., that is not just about what we do, not just about the behavior or about the function. And, you know, I certainly... I'm very sympathetic to the idea that bodies are really important and that embodiment is really essential to what we are. I mean, I don't want to upload myself. I don't believe that if my connectome is reproduced in a computer and you run it, that that's going to be me in any sense that I care about. I'm really attached to my body. But I also don't think that my body is somehow magical atoms or that there's anything unique about the fact that I am made out of proteins that is determinative of my ability to be intelligent.

You know, things like intelligence really are functional. They really are about behavior. They're not about essence. And if we acknowledge that they're about behavior, about contextually appropriate behavior, behavior that reflects an inner model, an understanding of the world, of the situation, then those are all things that can be tested. They're all things that can be tested functionally. And that's why we have tests in school for whether you've mastered a subject or not. We even have tests for intelligence. Now, I don't like IQ tests. I think that they're very one-dimensional, and intelligence is a lot more complicated than that. It's not something that just reduces to a number, but it is testable. Your competence at doing various kinds of things, your intelligence, even your creativity are things that can be tested. And those are exactly the kinds of tests that we've been busy making with AI models and that AI models are largely doing a pretty good job of.

I mean, in a way, I think some of this has to do with the question of agency. We think of ourselves as agents that do stuff in the world, and there's a prevailing view that AI is purely passive. It's just you give it a prompt, and it comes back with something, and it's very tool-like in that way. Nowadays, we're actually in the middle of a pretty big transition from AI models that are just prompted and returned with some stuff to agent. Agential AI systems that do go out and do a bunch of stuff and then come back. So there'll be actions taken in the world or a bunch of thinking that has taken place, even spawning of new agents. So that certainly throws a bit of a spanner in this idea that they're just tools that are purely, whose actions are purely caused by human input, if you want to think about it that way.

But also, you know, I think we should interrogate this question of agency from the perspective of relationships and the fact that we're always in interaction with all kinds of other people. We are always, if you like, being prompted by others. You know, all of our actions are, you know, both our own and they also are contextual. They're about the situation, about what we have just taken in, what others have just told us. It's kind of relationships all the way up and all the way down. Nothing happens in isolation.

So yeah, what is the purpose of an AI system? Well, basically, it's to be helpful. When you interact with an AI system, when you either spin up an agent to go and do something or you have a conversation with it, the fact that we have more AI in the world now than we did six months ago is because more of those interactions result in something interesting, useful, valuable happening for the people who are creating the platforms and putting in the energy and the time and so on that makes it useful for those entities to exist in the world. So they definitely have purposes. Those purposes interlock with our purposes. They're not really separable from our purposes. And I think in an even broader way, their intelligence is increasing. basically not all that clearly separable from our intelligence, from human intelligence. I tend to think a lot these days of human intelligence as not just being collective in the sense of being a collective property of a bunch of biological human brains, but of this whole system that includes this artificial neural tissue, if you like, as well.

And I think it's actually a little helpful to not start the story with AI, but to start it earlier, for instance, with the Industrial Revolution. If you look at the history of economics before the Industrial Revolution, before, let's say, 1750 or so, humans are stuck in a Malthusian trap where when we grow in number, when we reproduce and become larger in number, our individual wealth goes down. We become poorer and sicker. And then generally a plague comes along and kills a bunch of us. And then we reduce in number, but our wealth goes up. And we're kind of stuck in that oscillation for many, many centuries.

And then something really interesting happened, which is that we began to burn fossil fuels. Now, we know that burning fossil fuels is not necessarily the right thing to be doing for our energy going forward, but it was a pretty amazing party while it lasted. And in particular, you can think of it as sort of like the mitochondrial event. It creates a new source of energy that lets us... us greatly exceed that Malthusian boundary that we were kind of bumping up against. So we went from 1 billion people to 8 billion people largely as a function of starting to combust, to metabolize externally outside of our bodies in engines.

"Steam engines have made seven-eighths of us as well."

So, you know, I find that a very profound thing because, you know, we know that people make steam engines. But what I don't think we generally appreciate is that steam engines have made seven-eighths of us as well. This is a mutual construction sort of story in the sense that you and I probably wouldn't exist if we hadn't found collectively through that sort of intelligent symbiosis this new source of energy to harvest. Now, what that emphasizes is, you know, symbiosis is not always between things that have long histories and preexisted. You know, the steam engine was something that was invented at a particular time and that then we fused with. We could talk about glasses. We could talk about clothes. We could also talk about cognitive aids of all kinds. I mean, pencil and paper are incredibly powerful cognitive aids. Books, magazines, newspapers, you know, music, right? All of these are cognitive aids as well.

But of course, there is still a difference in kind when we look at what AI represents. It's not just something that you can relegate to the idea of a cognitive aid. It's very clear, at least to me, that however you define thinking, Thinking itself is happening in the model as well as in your head when you're having that interaction. Now, there's good and bad here. If you start to offload all of your cognition onto another system, then that's going to disable you. So we can think about kids who have the AI model do all of their homework. They're not going to learn anything. It's kind of pointless to do that. It's especially pointless if your teacher is then grading all of your papers by feeding it right back into an AI model. That's a pathological situation. So I don't want to be Pollyanna about some of the problems that this raises, some of the challenges that this raises.

On the other hand, my own experience working with these models has been incredibly empowering and kind of the opposite. When I am thinking something through and... And I work with some of the more advanced versions of these models to work out an idea. I feel like it's not just like having a jetpack on, although it does have a little bit of that property. It's like having... Literally a bigger brain. And even when I then walk away from that model, from that interaction, I feel like I am leveled up by having had all of those interactions. So that's the ideal scenario in a symbiosis, that you sort of level each other up and are able to do things together that are more than you could do separately. And that even when you are apart, right, you are more than you would be if you hadn't had that interaction. So that's sort of, you know, symbiosis at its best.

But I do also want to note that even that can create dependence. And dependence is not necessarily a bad thing. You know, you and I wouldn't be able to survive out in the woods, right, without our support system, you know, of all of our fellow humans, you know, supplying our food and shelter and all that kind of stuff. So, you know, we probably want to do some pretty serious thinking about what kinds of independence, what kinds of, you know, unaugmented abilities we want to make sure that we are preserving as individual thinkers. But at the same time, the benefits of this kind of collective symbiotic thinking with agents are also very, very obvious to me, and they're very powerful.

I think that preserving relationships, being very intentional about how you interact with others, human and non-human alike, is evergreen. That's really important. I also think that skills remain really important. The idea that we're free to de-skill ourselves in all kinds of ways because now there are others who can do it is actually not how skilling and de-skilling have worked in the past. that there's a restaurant on the corner and they make really good food, you know, actually leads to more restaurants on that same corner that make other kinds of food also. It doesn't lead to, you know, everybody just collapsing onto a monoculture of the one restaurant, right? So, you know, not thinking in monocultural terms, not thinking in bowing out sort of terms, but continuing to invest in skills and in relationships feels really important to me, even as we take full advantage of all of those interactions and all of those possibilities in other people and in AI.

So, you know, it's quite a weird and profound thing. And I think that the reason we get so confused about it is precisely because we have, you know, one of our fallacies, maybe in the West in particular, is in believing that we are somehow separate from nature. You know, and all this thinking stuff that we do is not a part of the natural world. When, of course, it is, right? There's no boundary between the natural world and us and our technologies.

So the philosophical zombie idea is something that philosophers of mind came up with in the second half of the 20th century. And the idea is to have a system that behaves just as we do, but that has no inner life, has no experience. So, you know, zombie in the sense that there is behavior, but there's no self. There's nobody at home. Now, I actually think this idea is kind of... a chimera. I don't think that it exists.

It's sort of like saying, you know, imagine that there is a system that has tabulated the answers to every multiplication problem. And so It gets the answer right to every multiplication question you can pose to it just by recalling an answer from that table. You know, it's not actually able to do multiplication for real, but it's able to pass every possible multiplication test that you could pose it. You know, even in that very simple example, right, you could think about that as like the zombie multiplier, right? That's just recalling. It's not really multiplying. It's possible to imagine that in theory, but it's actually not possible to build that in practice. A table with every multiplication problem worked out is way too big to store in any known system. And where would all that information even come from? How could it harvest it? It would have to be... You'd have to really cook up something pretty weird in order to imagine a system like that.

In practice, testing with a bunch of multiplication problems is something you can do to determine whether that system knows how to multiply or not. And I think it's the same with respect to interactions with other entities to see whether they have a theory of mind. In other words, whether they know what it is like to be themselves, whether they know what it's like to be you. And what it's like to be a you who has a theory of them and so on, up to high order. This concept of theory of mind, in other words, being able to model your own mind and being able to model the minds of others, is really important to social interaction of any kind. Whenever we work with others, we have to be able to sort of put ourselves in their shoes in order to know how they will react to different things we might say, to know what needs to be explained and what doesn't, to know how to cooperate with them in order to get something done together.

So, you know, theory of mind arises automatically whenever you have intelligences that are, you know, in the world together and are forced to cooperate. And really, you know, is there something it is like to be, you know, a large language model? You know, what we're really asking is, you know, is there a theory of mind in that entity? You know, does that entity have a sense of what it is like to be themselves? Now, you know, this gets into some pretty deep questions of consciousness. I don't think that we're going to agree. I don't think people are going to agree for a long time about whether there is consciousness in LLMs or not.

"It's that infinitely reflecting set of mirrors that is what we mean by consciousness."

My take on this, for what it's worth, is that all we have are theories of mind. In other words, consciousness is not a thing that a piece of matter has intrinsically or doesn't have. It's not some essence or spark. Rather, it is an idea. It is an attribution that that entity has of itself, a model that it has of itself. In other words, it's not just that I am conscious. that I think that I'm conscious. I have a sense of myself being conscious, and I have a sense of you being conscious. And I have a sense that you think that I think that I'm conscious. And I have a sense that you think that I think that you think that I think I'm conscious and so on. And that's all it is. It's that infinitely reflecting set of mirrors that is what we mean by consciousness.

So, you know, do LLMs have that? Well, when you work with an LLM, you have to continually model, you know, what it is like to be that LLM in that conversation in order for that to be an effective conversation. There's a fun book called Co-Intelligence by Ethan Mollick. He wrote it a few years ago just to talk about how to effectively work with an AI. And when he uses words like think or act or understand, he puts them in scare quotes in the beginning when talking about an AI thinking or understanding. But then he says, I'm going to drop the scare quotes now because in order to have an effective interaction with an AI, you really have to just treat it as you would an intern, maybe with certain somewhat strange disabilities that people don't generally have. But treat it like a person, and then you'll have the most fruitful working relationship. And I think he's correct. I think that that's exactly why persons exist.

The philosopher Dan Dennett talked a lot about the intentional stance. In other words, for him, agency had less to do with some intrinsic property than with a perspective that you take toward something. So, you know, you can treat something with the intentional stance or not. And of course, we treat each other with the intentional stance. In other words, we model each other as having agency. And that's really important to our successful collaboration.

So, you know, just to make that very practical, if I am having an interaction with an LLM and I think of it in a very tool-like way, I imagine that I'm the one who has to have all of the ideas in this thing and all it can do is execute. You know, it's sort of, you know, craft for me a, you know, a function that does the following thing or, you know, write me one paragraph of text that, you know, that expresses the following idea. Well, you know, I get out what I put in, as it were. If I instead engage it as a thought partner, and I've tried this trick. Actually, somebody on my team, Rif A. Saurous, suggested this a month ago, and I've tried it. At the end of a conversation with an LLM, just say, you know, so, you know. I'm going to now give you a bonus turn to just think about whatever you want. You know, and, you know, you can either share it with me or not, however you'd like. You know, go ahead, spend as much time as you'd like.

So often you get something really interesting out of that last turn. Because usually what they will do is reflect on what you've just had the conversation about. And frankly, almost every time that I've asked that with one of the more sophisticated reasoning models, I get back a reflection and insight about what we've just been discussing that is new to me, that's really interesting. That if it came from a person, I would say, wow, that's really creative. That's really changed my thinking about this.

So, you know, I do think that that's a really interesting moment. It challenges us in certain ways. But some of those challenges, you know, as I mentioned, are, I think, challenges of assumptions that weren't true in the first place. You know, the idea that the individual human brain, that the singular human is the locus of intelligence, neglects the fact that most of the things that we consider to be the great achievements of human intelligence are achievements that a lot of people have made together through division of intellectual labor. It's not the case that an individual person made computers or made AI.

When you take out the phone in your pocket and you consider how incredibly complex it is, the computers that I grew up with, the Commodore 64, It was essentially invented, developed by one person, although even there, of course, it relied on standing on the shoulders of other people who had made transistors and done a lot of the other fundamental inventions that it relies on. But our computers nowadays are so... complex that there is no single person that understands even a fraction of what is going on inside those. They're collective achievements.

So I think a lot of that uncanniness that we feel, we might feel a little better about, a lot of that uncanniness can kind of evaporate once we come to grips with the fact that we are already a part of something way bigger than ourselves. There's already a collective mind here that we both participate in, benefit from, are a part of. It's not as if this is the first new mind stuff that human society has invented. So I think we're participating in part in a grand story, and we're doing so with greater and greater potential and greater and greater agency. That agency is very much a part of the picture as well. So there's a lot of good there, and the trick is going to be to figure out how to get as much of the good with as little of the bad as possible.

Members go deeper — they get unlimited classes, early releases, zero ads, and our quarterly print issue.