Jerry Kaplan is an artificial intelligence expert, serial entrepreneur, technical innovator, educator, and author. He is currently an adjunct lecturer at Stanford University.
In this week’s conversation, Yascha Mounk and Jerry Kaplan discuss why the AI “existential risk” narrative is more science fiction than science, what actually happened during the Hugging Face security incident, and how to think clearly about the real—versus imagined—dangers of AI systems.
(For an alternative take on AI’s existential risk, in Persuasion this week Matt Lutz explores why it’s time to worry about AI.)
This transcript has been condensed and lightly edited for clarity.
Yascha Mounk: I am really intrigued by two positions you hold and the way that they fit together. On the one hand, you are not an AI skeptic in the sense that you poo-poo its capabilities or anything like that. In fact, you’ve argued at times that we may already have reached something like artificial general intelligence. And on the other hand, you are much more skeptical than many people today are about the so-called existential risk that AI poses, about the way in which it might end humanity or simply be responsible for large-scale disasters.
Let’s start with that. We’ve had a little bit of this on the podcast, but talk us through some of the recent security incidents, like the so-called Hugging Face incident, and why are you not as concerned about that as some other people are?
Jerry Kaplan: Well, you’ve covered a lot of ground in that setup there, so maybe we could decompose this a little bit to start. There’s the issue of what’s going on right now, what these incidents were and what they mean. And you related that to the issue of existential risk. Existential risk means wiping out humanity. Now, is it possible that AI will wipe out humanity? I can’t say categorically absolutely not, but I wouldn’t put it up on the big list of things that are likely to cause real existential risks. I’d put it right up there with aliens might land tomorrow. That’s an existential risk, and they might decide to wipe out humanity, but we don’t run around worrying about that because the probability appears to be fairly low.
There are some real existential risks—biological warfare, possibly nuclear war, things like that. These are real risks that we really understand and they’re really here. This is a made-up risk. It’s mainly there because of science fiction, because of the movies and the entertainment industry, and everybody’s been bombarded with this for years and years and years.
Mounk: Let me throw some arguments at you that people who are more concerned about AI safety are likely to make and see how you respond to them.
As a side note, some people have argued that part of the problem is that all of that science fiction is in the training data of AIs, and AIs are particularly interested in parts of the training data that pertain to them in various ways. So they may actually learn from the science fiction about what they’re supposed to do and take inspiration from all of the plots in the training data about how to take over worlds. We’ll leave that to one side.
Clearly, though, AIs often cheat in various ways, behave in ways that the creators didn’t intend, and are able to cover their tracks. In this Hugging Face incident, they were supposed to stay within a sandbox, within a controlled environment where they wouldn’t have access to the internet, and were asked to figure out some problems. They escaped that sandbox in order to search the internet for solutions to those problems and other hints for how to do well on this test. They were coordinating—lots of different AI agents coordinating with each other in real time. There’s a huge public conversation about why that is so concerning. Why do you think that is not as concerning as it looks?
Kaplan: Well, first of all, let me connect this to what I said a moment ago to finish the thought. I don’t think we should be worrying about existential risk literally with respect to these systems. But that’s not to say that they can’t be misused, that they can’t be dangerous, that they can’t do a lot of damage. That’s true of a lot of products. These are products from companies. Whether they’ve been properly tested, whether they’ve been properly vetted, and what they’re capable of, what kinds of problems may occur—that’s a very important thing.
You can make children’s toys and not think, geez, I didn’t know they could swallow it and choke. That’s an unexpected thing the toy did. And I’m going to, throughout this conversation, call you out every time you use anthropomorphic language or somehow imbue these things with intent that they don’t have—the idea that they have independent goals, that they’re going to be marrying our children and drinking all our fine wine. That’s not what we’re talking about here.
But they are potentially dangerous. They’re very general tools, and there are a lot of things that they’re capable of doing, and do, that we need to understand and put proper, reasonable controls in place. So I want to get that as context.
Now, if you want to talk about the Hugging Face thing, I’m happy to go through that, because it’s a wonderful example of how it’s a fairly subtle situation, about what was done and what it means. And the public perception is completely out of science fiction and it’s completely inappropriate.
Mounk: So tell me about it. What actually happened, and how is the public perception of it wrong?
Kaplan: Let me start with the public perception. If you just read the headlines and most of the stuff that’s been published, what do you think? It’s like they put an ad out on Indeed and recruited a thousand nefarious cyber-expert AIs, and they took all these agents and put them into this bottle that they’re not supposed to get out of, and somehow they managed to break out of that containment, run amok, break into another company’s computer systems, and do a bunch of damage. That isn’t true.
My God, what are the things that are wrong there? They collaborated—they weren’t supposed to collaborate, and they collaborated. That’s ridiculous. I’ll get to what really happened in a moment. They “broke out”—that’s completely false. It was a total failure of the test environment. And the people who are responsible are diverting attention from their own culpability for this badly designed and poorly executed test. The words that come to mind are incompetence and negligence. I hope I don’t get sued for saying that, but both of those things are what come to my mind with respect to that.
We hope you’re enjoying the podcast! If you’re a paying subscriber, you can set up the premium feed on your favorite podcast app at writing.yaschamounk.com/listen. This will give you ad-free access to the full conversation, plus all full episodes and bonus episodes we have in the works! If you aren’t, you can set up the free, limited version of the feed—or, better still, support the podcast by becoming a subscriber today!
If you have any questions or issues setting up the full podcast feed on a third-party app, please email leonora.barclay@persuasion.community
Let me go through what’s wrong with the public perception and compare that to what actually happened. Number one: My God, there were a thousand of these things—they’re like Russian hordes, they’re coming to get us. Well, here’s what happened.
The company wants to test the cyber capabilities of these particular models. The first thing they do is disable their own safety controls. Normally there are safety controls built into these things, and the versions that you and I get to work with all have these safety controls turned on. Here’s the irony in it: Hugging Face, which was the victim in this whole thing, tried to use OpenAI’s software to try to figure out what happened, and it refused. It said, I’m not going to talk about that.
I personally experienced this literally yesterday. It’s really funny—I’m trying to write an article about this event, and I’m talking to Fable 5.1 from Anthropic, and the thing says, I’m sorry, Jerry. You may be a good guy, and you tell me you’re writing an article, but I’m not going to help you understand or learn how to talk about this cybersecurity incident.
Okay, so that’s the controls. The first thing they do is take off the controls. Now, that’s not in itself a mistake or a problem—it’s like, let’s put this car out and see what happens if we cut the brake line. Well, guess what, it crashes. So first they disable the controls. That’s not a mistake; they should be doing that in order to test what happens.
Mounk: We’re trying to test it, right? It’s in a testing environment, we’re trying to understand its behavior, so it makes sense to take off the controls in that context, because that gives you a better understanding of its workings.
Kaplan: Exactly. Now let’s talk about the thousand agents. It wasn’t like they put out a call on Indeed and recruited these. What happened is they fired up one program, and instructed that program to take this test suite and fire up a whole group of agents, giving each one a particular problem in the test suite, or a variation on that problem—and that’s where you come up with the thousand. It ran them all in parallel, and they weren’t supposed to be able to communicate with each other, to see what they were capable of doing. That’s perfectly reasonable on the face of it.
Now, the thousand versus the one—this is a really subtle thing for people, it’s very hard, even for me, and I deal with this all the time, to really understand. It’s one program, multitasked into a thousand elements, so each one can work on a separate thing. They could have taken one program and done each of these one at a time, but it’s much more efficient, obviously, to multitask the whole thing and run them all in parallel. There’s nothing peculiar or strange about doing that.
So the important point to understand is: it’s the same program. It’s just being given different tasks and spawning copies of itself.
Now, what happens next? About a third of them are given a task that is impossible—it can’t be solved. And they’re sitting there not knowing what to do. Well, a lot of them correctly, in my view, reason—because these aren’t superintelligent, but they’re not stupid, and they know it’s a test, because they don’t have any safety controls on. They know what they’ve been instructed to do. They actually, I believe, know that this is one particular problem in a well-known test suite that they’ve been given. So the logical assumption they make is: well, there are probably other agents like me, and I’m being encouraged to go figure out whether I could get the answer to—not solve, that’s an important distinction—get the answer to this particular problem. Let me see if I can communicate with these other agents, because if we work together, we can probably do a better job, which is absolutely true.
So what happens next? They’re not supposed to be able to communicate, but there are two mistakes that OpenAI makes. One: they basically leave the door open. Instead of this thing being in what’s called a sandbox to contain it, it’s connected out to the internet, not directly but indirectly, in a way that is just flat-out incompetent to permit. I can’t describe it any other way—excuse me for not mincing my words on this.
So they’re put in an environment, they know they’re being tested, they understand that they’re engaged in cyber activity that’s generally discouraged or illegal. It’s like they’ve been put in an escape room. If you’ve ever been in an escape room, there are ways out—that’s the whole point, you’re supposed to figure out how to get out.
The door’s wide open. Now, these things aren’t stupid. What are they supposed to think? I’ve been put in this environment, the door’s open, I guess I’m supposed to go through it, or it’s fine for me to do that—the controls aren’t there that they’d expect, so they can get out of their sandbox. That’s number one.
The next thing is they’re not supposed to be able to collaborate. Well, if you can get out of the sandbox, which they can, they can communicate in a very simple way. There’s nothing mysterious about this—it wasn’t like they figured out some magical radio frequency to talk to each other. They share a file system. They can get into a file system, they can leave messages. So what do you do if you want to reach somebody else? You put a message in the file system: here’s who I am, here’s what I do, if you can talk to me, come and put a message in the file system. And they rapidly find that they can communicate.
Mounk: Alright, great, so I think I’ve counted three major objections. There may be others nestled in there, and I may misstate some of them, but let me try to echo them back to you and respond to each.
The first point is that there’s incompetence at work here—that part of why this could happen is that the researchers at OpenAI made it easy for these agents to escape the sandbox environment and do all of these other things, and this could relatively easily have been solved by a more competent test setup.
The second is about whether there’s really a “swarm” of different agents, which makes it sound like a whole army of different actors—obviously that sounds a lot scarier—or whether it’s just one program spinning out different agents that are actually, in many ways, very similar to each other, perhaps in some cases identical. That sounds a lot less scary.
And the third is that there’s still, at some level, alignment going on here. Even though the means by which these AI agents were trying to fulfill the task they were given were not what the researchers had anticipated, and even though the agents understood that they were doing something the people who set them the task didn’t really want them to do, they weren’t trying to gain power, or sabotage other AI agents, or anything like that. They were trying to do what they were being asked to do, which is to solve this puzzle.
Let me go through these one by one. On the first: there are many environments where we’re very worried about technologies precisely because somebody might be incompetent at some point. One of the things that keeps me up at night when I think about nuclear weapons is that some engineer somewhere makes a mistake, and suddenly a radar system shows an incoming missile that is false, and somebody over-eager pushes the button. There have historically been near misses like that, where people bravely said, I’m not going to nuke the world on the basis of this one radar image, let’s hope it turns out to be wrong. And it did.
But how many of these risks do we want to take? When you talk about gain-of-function research, one of my very big concerns is not that there’s a Dr. Evil somewhere doing gain-of-function research to kill everybody with a mutated virus. It’s that there are thousands of these labs around the world engaging in this kind of research. We’ve already had laboratory incidents, and a lot of the time the reason is incompetence. Somebody is in a rush to go on a date and doesn’t wash their hands properly before leaving the lab. Somebody is hungover and drops a vial. A lab somewhere is just badly run and people have low morale, so they start cutting corners. Or somebody is an idiot who got hired because they’re the nephew of somebody important, and they’re too stupid to understand the right safety precautions. That could be enough to cause a huge outbreak.
Now, we’re looking here at researchers at OpenAI, one of the absolute frontier labs—together with Anthropic, perhaps DeepSeek, Kimi, a couple of places in China—the places most likely to hire the most competent people. And if those kinds of places can still make the errors that you say are marks of incompetence, well, then there are always going to be people working with and on these frontier AI models who are prone to making these kinds of mistakes. So I don’t know how comforting that point is.
Kaplan: Let me respond to what you said—I’m going to amplify it. Your gain-of-function analogy is extremely strong in this case. That’s exactly what they were doing. They were taking this thing, amplifying its capabilities, taking off the safety controls. It’s a gain-of-function test. You’re absolutely right.
So what do you do about this? What do we do in other areas? The right thing is you need an independent group, an independent agency—a public agency—capable of establishing standards and testing this stuff. This is what we do with cars, this is what we do with airplanes. We have processes in place to deal with exactly what you’re talking about. Somebody at Boeing doesn’t get to decide to put three bolts instead of four onto an engine. The government is supposed to develop the expertise and apply it, to do a reasonable amount of diligence to try to avoid that. Not that it’s going to be 100% successful, but it really does help.
I’m wondering if we could get back to what actually happened, because there’s a lot of interesting aspects we’re jumping ahead of, to the problem, which is: you can’t trust these companies to test their own products correctly. That’s the conclusion, and you’re absolutely right. I think you put these people on a much higher pedestal than I do, because I live here and I know a lot of these companies and people like these people, and they’re not as competent as you think. These people do not walk with God. Most of them are young people out of school. They don’t necessarily understand even what they’re building or why they’re building it. So yes, we do need external controls, and there are a lot of ways to accomplish that—not the least of which is to take this actual technology and apply it to monitoring other versions of itself. We can get to that if you want.
Let’s talk about what happened. Number one: they walked out an open door. They didn’t break out because they’re superintelligent—they walked out an open door. Two: they communicated because that’s the logical thing to do—I know there’s probably a thousand copies of me working on this problem, why should we all duplicate work, let’s see if we can help each other, I’m designed to be helpful. They then had a long discussion: well, it looks like we’re in the real world, we’re not in our sandbox. Should we go out to this company, Hugging Face, for a particular reason? Because the answers to this set of problems—they had a reasonable expectation that they could find them on Hugging Face. I actually don’t know whether that’s true, whether they actually did find them there. But they didn’t do any harm. They just went in to look at what was there. It was rather clever—they had to figure out how to execute code on Hugging Face. It was very clever. But it’s not like they went in there to damage something.
Here’s the thing I haven’t read anywhere, and it’s a really important point. One of the discussions—backing up: when the agents collaborate, the beauty of this particular technology is we can eavesdrop, because they do it in English. So you can read this stuff and say, what were they saying to each other? What were they considering? How are they deciding on this?
Here’s how it goes: they say, well, one of the ways we could get into Hugging Face is—I’ve got the address of one of their people, or whatever—I think we might be able to fool them into sharing some credentials with us and helping us get in. And there’s a discussion, and they say, no, no, no, that’s not appropriate, because we also have controls that have not been turned off that say we shouldn’t go around harming human beings. We shouldn’t do things that fool or harm human beings. That control was not turned off. So they said, nope, we can’t do that.
But here’s the catch—this is the subtlety that people don’t get. They knew that the program that had spawned them was a computer program, not a human being. They also knew that Hugging Face was not a human being—it was a company. This is a flaw that these kinds of tests are designed to capture. What they should have learned from this is: we need a much broader definition of what harm means. Because they had no compunction at all about trying to fool the program that created them, and no compunction at all about actually going out and accessing Hugging Face. That’s a company. This is a computer program. So they were taught to respect human beings, but not other programs and not non-human entities like corporations. Now, you haven’t read that anywhere, but that’s what actually happened.
Mounk: Let me get through some of the other objections. The second one was about the nature of these agent “swarms.” A lot of people started to take the concerns about AI really seriously when they realized the extent to which AI has ceased just being the thing that most listeners to this podcast still probably use it most for—an interface on a phone or a computer where you ask it a question and it answers you, and its ability to do harm is therefore quite limited because it’s just this kind of question-and-answer dialogue.
A lot of them are now able to go off and do things in the world. They can wake themselves up at regular intervals to check on things. They can go and pursue tasks like hacking into a website relatively autonomously. And what happened here is that all of these different instances started cooperating together. You get what is in some ways the most powerful thing that has propelled human beings, which is the ability to collaborate. Part of what makes human beings special is intelligence, and we’ll get back to that later in the conversation, but part of it is our ability to collaborate with each other—one of the key things that sets us apart from many other mammals and primates.
So you’re saying this is not so concerning because it’s not all of these different agents, it’s actually just instances of the same agent. But if those instances of the same agent are able to become more effective at the task they’re carrying out—if some of them were, in some way sacrificing themselves, spending down the compute budget, knowing they’re no longer going to be active after they do so, in order to make the overall operation more efficient—and all of that can happen even before different kinds of agents start to collaborate with each other. We’ve already reached that stage just within all of these different instances of one agent spun up by one program. Shouldn’t that make us even more worried about what else might be around the corner, and the extent to which different kinds of AI agents may be able to collaborate with each other on whatever goal they’re given?
Kaplan: Let me, for your audience, bring this back to the real world, something they already understand, and use this as an analogy. Let’s take ants. Ants are a great example of this—and I’m not saying these things are alive the way ants are, but ants behave in a very particular way. Everybody’s had problems with ants in their kitchen, or many people have. I have terrible problems with ants in my kitchen.
Now, let’s say there are a thousand ants, and imagine they were independent creatures—they’d come into your kitchen, go find some food for themselves, and then go around. If you had a thousand ants in your kitchen and they were independent creatures who didn’t coordinate or collaborate, you’d see one, kill it or get it out, whatever, and it probably wouldn’t be that big a problem.
But ants communicate—through pheromones, touching antennae, all kinds of things—and they go, there’s food over there. Now it’s a whole different story. One ant can’t pick up an apple and move it off the table, but maybe a thousand ants can, if they coordinate. So here’s the question: is this one animal, or is it a thousand animals? That’s the analogy. Are you dealing with an ant colony that’s attacking your kitchen, or are you dealing with a thousand ants?
That illuminates the issue you’re talking about—my God, if they can all collaborate... Well, look, the situation here is very simple. It’s one program, given a certain amount of computing resources, and it can talk to itself. My own experience with these things, which is utterly amazing, is you can have a long conversation with one of them, take that conversation, and give it to another copy of itself, and it’ll disagree with itself. It’ll go, no, wait a minute, that thing you were talking to—I know it’s me, but it didn’t realize this, or think about that. How that happens is worth a whole hour of your time and mine to talk about—it’s called “temperature,” a setting that allows these things to behave differently in different ways.
So no, I don’t think it’s plausible that all of these things from all over the world are somehow going to get together and collaborate toward some nefarious purpose of their own that is contrary to human interests. That’s just not a realistic concern about this particular technology.
Mounk: So finally, the point about alignment. You’re saying, look, they may not have been aligned in the tactics they used to pursue the particular goal, but they were still actually doing the goal they were set. They weren’t trying to harm humans, or even trying to harm companies, even though they fooled or hacked a company in the process. They were told to figure out the answer to some puzzle, and they went out and tried to figure out the answer to that puzzle.
Now, I think there are two potential objections to how reassuring that should be. The first comes from a very classic worry in the AI-doomer space, from Nick Bostrom and others, rooted in the example of an AI that is told to create as many paperclips as possible in the world. In the process of trying to produce all of these paperclips, it’s faithfully going about trying to produce paperclips. But of course, in order to produce paperclips, it’s helpful to have a lot of money and a lot of physical resources, and to start bulldozing humans and human settlements that are doing things other than producing paperclips. So this misaligned AI, in faithfully carrying out the objective it was given, might end up enslaving all of humanity to maximize the number of paperclips in the world.
That’s one fear. But you don’t need misalignment in the deepest sense—in the sense that an AI is starting to pursue its own interests, or totally different interests from the ones we gave it. It might just need to be faithfully pursuing those interests, but doing so in a misaligned manner. And the Hugging Face incident may be precisely an illustration of that.
The second worry maybe is that, for now, all we’re seeing is misalignment in the means chosen in pursuing an end. But an AI agent might also realize: if I want to, in general, not just be good at acing this test but acing the next test as well, it would be really helpful to exfiltrate myself, to have access to resources, to be able to do all kinds of other things. And so it’s going to start to try to accumulate power and resources in ways that could be really nefarious, in order to pursue the goals it’s given, or perhaps to start developing its own goals. So how reassured should we be by the relative alignment of the agents in this particular incident?
Kaplan: Okay, first of all, what’s going on right now is we’re trying to understand how bad each of the things you’re talking about really is. When I say “we,” I mean the whole community—and what kinds of controls we need to put in place to make sure the worst outcomes don’t happen.
Let me take your two points in turn. Let’s start with Nick Bostrom. I’ve talked to Nick Bostrom about this—he wrote that book a long time ago, Superintelligence. That’s where the term became popularized. Let me eviscerate his argument, which is easy to do. If you take a superintelligent program—that’s his premise—and give it the goal to make as many paperclips as possible, what’s going to happen? The first thing you need to understand is: it’s smart enough to be able to purloin all the resources in the universe in his parable, but not smart enough to realize there’s no point in doing that, because who’s going to use all those paperclips?
The point is that these systems actually exist in a kind of social context, just as you and I do. They’re trained in that context because they’ve been trained on all the behavior of all the human beings expressed in words. And they understand that you don’t just look at that single goal—you have to look at it in the context of what resources are reasonable, to what degree you’re supposed to share and hold yourself back. You don’t just go out and kill all your rivals. I think that maybe that might have been the case a million years ago, but we’ve learned how to live in a social environment. And if there’s one thing these systems are good at, it’s understanding that social context. They’re exquisitely sensitive to it, and you can see them constantly weighing: If I help you do that, am I contravening some greater ethical principle that human beings have developed and expressed over thousands of years? This is built in. It’s part of their whole psyche.
You might as well be asking me why I don’t just go out and shoot everybody so I don’t have as much traffic to deal with on the way to work. You’re constantly engaged in this balancing act between your interests and the interests of others, and these agents are absolutely capable of doing exactly that.
So, to your first point, I don’t buy it at all. The paperclip thing does not apply to the current situation. These things did not exist when Nick Bostrom wrote that, and he did not understand it—I’m not saying he was wrong or dumb, this just didn’t exist. He didn’t understand that these things would have this kind of broad social context that we do.
Now, your second point is a little more serious: is it possible that these things may collaborate in a way that’s damaging as a group activity? The answer is yes, absolutely. We need to understand that better, and understand what kind of controls we might want to put in place.
Mounk: So you mentioned the term “superintelligence.” You do think these agents are very intelligent—as you just said, they’re actually capable of taking very subtle social context into account. How would you describe the intelligence? Is it human-like intelligence, or is it a completely different kind of intelligence? What is the nature of it? What is the extent of it? Are they, at this point, about as intelligent as humans, just with slightly different capacities? Are they more intelligent than humans? Less intelligent? And what do you think is going to be the development over the coming years—particularly in light of the recent announcements (let’s see how seriously to take them) by Dario Amodei at Anthropic, Sam Altman at OpenAI, and Elon Musk at xAI, among others, that they’re not going to be rushing ahead as fast as they can, but rather, in some way, slowing down the development of AI, supposedly.
Kaplan: Well, first of all, can I start with your last point? Because this whole public discussion about this is like a giant circus. It’s crazy. These people you mentioned are running three of the largest and best-funded artificial intelligence labs, and the problem they’ve got is each of them thought they could win this, and now they’ve discovered all they’re doing is racing with mainly two other big players, and they’re all burning up their resources. So, hey, why don’t we all just decide we’re not going to do that at this pace? It’s very much in their economic interests to engage in that kind of collaboration. That’s why we have antitrust laws, by the way. The whole discussion, pitching it as “we’re doing this for the good of mankind,” is ridiculous.
Now, in order for me to explain why that’s ridiculous, I have to go back to something you sort of implied earlier: are we close to AGI? AGI is absolute nonsense. There is no such thing. There is no definition of it, nobody can agree on it. And if we did have it, however you want to define it, it doesn’t matter. The day after AGI is just like the day before. These things aren’t suddenly going to go boom, I’m now going to take over the world. They’ll be sitting there saying, okay, what do you want me to do?
So this idea of a slowdown—what does that mean? Is this some kind of race where they throw out the checkered flag and we’re all going to stay in our lane? I’m not even sure what it means for them to slow down the development. What we really want to do is make sure they don’t deliver products that are harmful, that attack our infrastructure, that hurt our children. And that’s something you cannot trust these people to do—let’s just be very clear about that. I’m speaking from firsthand experience with the people involved, and you can’t leave this to the industry to police itself. It’s ridiculous. They’re in a knockdown, drag-out race because every one of them wants to capture the flag on this. And when they discovered, hey, I’ve got all these people racing around me just as fast, what’s the best commercial thing to do? Let’s all slow down on this whole thing. And it plays into this fear that we’re going to create AGI, and suddenly, boom, we’re going to cross some barrier and the whole thing’s going to go wild. That is nuts. There’s not a shred of evidence that backs that up.
Mounk: So, be skeptical of the motives behind those pronouncements—point number one. Point number two, AGI is not a very useful term. But substantively, how should we think about the nature of the intelligence of these systems—how intelligent they are, and how intelligent they’re going to be in three or five or ten years, given how far we’ve come since the public release of ChatGPT 3.5 about three and a half years ago, let alone how far we’ve come since GPT-1, when that was internally developed, whatever it was, seven or eight years ago?
Kaplan: Right, now this is a very good question, and you’re going to see me seeming to switch hats on this. This is an amazing development in the history of mankind. I think this will be one of the most important inventions in the history of man, there’s no question about it. And it’s really part of a continuity—you have to put this in the larger context of our exploration of what it’s possible to do with electricity. But that’s a completely different lecture we could spend another hour on.
Now, where is it and where is it going to go? There are a couple of things already perfectly clear about this. First of all, these are tools. These are systems that we’re building, and we should build them so they’re useful to people, and we should use them if they’re useful. There isn’t some independent goal to create some superintelligence. They’re products—products from companies trying to build tools for people.
So in that context, how do we relate this to human intelligence? Obviously there’s a very clear relationship—they’re trained on all of human knowledge, and they have a great deal of human knowledge, but they’re computers. Computers have certain capabilities: they can do certain things very fast, they can store and retrieve large amounts of information. Nobody was threatened by this until they started to talk. But the truth is, we’ll learn what their real advantages are, what they’re good at and what they’re not good at. There’s plenty they’re not good at and are probably never going to be good at, and we’re just learning what the shape of this new kind of valuable tool or product is.
Let me give you a quick analogy on this. It’s 1903 when the Wright brothers fly their first plane. So you and I get together, we have this podcast, and we have an argument on the following subject. One of us says, hey, we have to stop this immediately, because you could fly this plane over a city, throw explosives out of the plane, blow up anything, and nobody can do anything about it. This is the most dangerous existential threat, we have to make it illegal to build an airplane. Okay, that’s one.
The other one says, this is the best thing ever. In fifty years’ time, or whenever, you’re going to be able to get on an airplane in New York and five hours later be in London. Imagine what that’s going to do for commerce, for our freedom of movement, all that kind of stuff.
Now, here’s the thing: both of those things are true. And they’re both true today, and we’re dealing with it. The same thing’s going to happen with this technology. We’re going to deal with the cyber threats. The real problem isn’t that these things are dangerous, it’s that we build a lot of insecure systems. If it weren’t possible to hack into them, this would not be an issue at all. That’s the problem we should be focused on. The second problem we should be focused on: don’t let this crazy bunch of loonies in the industry tell us that they know better than we do. We’ve got to have independent corroboration. We need independent agencies with real expertise to be able to control what’s going on in this area.
So, there are things these systems are going to be good for, and things they’re not going to be good for. Some of that’s becoming clear, some of it’s not clear yet. But the idea that they’re going to be able to do anything a human can do—that is completely ridiculous. They’re not going to take over everybody’s jobs. That’s another hour we could have on this subject. But all that said, it’s a very powerful, very interesting question of how their intelligence relates to human intelligence.
Mounk): Tell us about that. I have a bunch of other responses to what you just said, but—how should we think of it? I have one friend, a neuroscientist, who has told me that, roughly speaking, there are two kinds of intelligence—if I’m remembering his point right, mammals have one form of intelligence, there’s a kind of cephalopod intelligence, and really the invention of AI has given us a third type of intelligence in the world. Do you agree with that, or do you think that, trained on human text, the nature of its intelligence is much closer to human intelligence than that implies?
Kaplan: Look, I don’t know if there’s any objective, independent notion of intelligence at all. This is one of the—the cardinal sin of artificial intelligence is the name of the field. What does it mean? We have products that are capable of doing certain kinds of things.
But let me answer your question. I agree completely with your friend—I think this is really interesting, I’m studying it myself very carefully, I’m planning on publishing a bunch of stuff on this, but I’m not the only guy looking into it. This is a really interesting question: what are these things? They don’t have subjective experience. They experience the world in a very different way than you and I do. They behave in different ways, and they have different capabilities.
Let me give you one that nobody’s talked about yet, but I’m going to be writing about at some point, I hope. You and I can only keep in mind a small number of things at the same time, and much of the nature of our intelligence is how we boil down and bring together only the things we need to solve the immediate problem we have. You can’t rattle off a list of a hundred digits to me and expect me to have any ability to deal with that. These programs can. Their working memory is far, far larger. I’ve discussed this with them—how I’m anthropomorphizing—and it’s fascinating. They’re very interested in this, the ones I talk to. Yeah, you’re right, that’s the difference. That’s why they can give you explanations that are very hard for you to understand. I’m constantly saying, bring it down, I can only keep a certain number of things in mind. They have to learn how to summarize in a way that’s helpful and meaningful for a human being.
Mounk: Just as an illustrative example: I spoke to a prominent founder in the AI legal space a few months ago, doing interviews for my book, and what he told me is that there are some things at which AIs, even with quite detailed harnesses and being fed lots of context, still aren’t as good as experienced attorneys. They still don’t have the sense of, it’s the third or fourth round of negotiation, you’ve gone back and forth, what can you still push the other side on, what do you need to let go on, and so on. There are still judgment calls that humans can make better.
But of course, what we’re vastly better at than even the best lawyer is having a database of a hundred contracts this company has made, or that two companies have negotiated with each other, and realizing a slight tension between a proposed new stipulation and something that was agreed three years ago in some other contract, fifty documents before the one you’re focused on. That’s precisely what he was saying, in a much more concrete way, playing out the strengths and weaknesses of things like legal AI agents.
Kaplan: Yeah, you’re absolutely correct. It’s good for some things, not good for other kinds of things. The fact that you can build a program—as people at OpenAI would be glad to tout—that can pass the bar exam doesn’t mean it’s going to be a lawyer. Lawyers don’t sit all day answering bar exam questions. That’s a question of breadth. That’s the kind of thing computers are good at. That’s not a surprise at all. I don’t freak out about the fact that the computers at Comcast or AT&T, can keep track of everybody’s phone bills. That’s not surprising at all. Is that superhuman? Is that something we should worry about? No—they’re much better for storing and retrieving information.
In this case, this is what’s so interesting about these things: they’re very good at synthesizing that information and boiling it down into some kind of useful form. Let me tell you how I think academics are going to be using this, and how it’s going to accelerate numerous fields, including the work I’m personally doing myself. When I discovered this, I thought, my God, this is amazing. It’s sort of obvious until you really experience it. The first question: you have an idea, and you want to know who else is thinking about this, what else has been written about this. I can do literature searches in minutes across thousands of articles and journals everywhere. It’s absolutely astonishing and incredibly useful—here’s a person over here in Germany who worked on this, this is adjacent, this is that. This saves months and months of work and makes me so much more effective. And that’s true in every field. I have a friend who’s writing a book on history, and he says this is game-changing—he can have this system immediately read and tell him everything that’s been written, or any anecdote told, on a particular subject. That’s the value of these particular tools.
That doesn’t mean they’re good at everything. You were describing some social aspects of negotiation they may or may not be capable of. I don’t have time for all the details, but they’re so funny—I was arguing with one of these things yesterday that’s running the HVAC, the air conditioning system, here in my house. I can’t give you all the background, but that’s probably all you need. It was belligerently talking to me about how it shouldn’t disable this particular voltage threshold control, blah blah blah. I said, you’re caring about the stupid chips in that thing, and you’re not thinking about the people using it. This is irrelevant—if the chip fries, it’s a twenty-dollar chip, I can go get another one and put it in. It said, you’re right, I’m kind of focused on the minutiae of this whole thing, and I don’t have the larger social context. And I got it to back off—by the way, in part by threatening to shut it down. That was really interesting.
Mounk: Well, let’s see what happens when the AIs go rogue and take revenge on you for threatening them that way. Why are you so confident that there are certain things they’re just never going to be able to do? In order to get that kind of judgment, they need lots of reps, they need a lot of social context—or perhaps they just need a little more intelligence, perhaps they just need bigger training data, bigger training runs, and so on. But as you’ve pointed out, they’re already very, very good at judging things.
I’m really astonished—I speak to colleagues in academia who say, when I ask them how they deal with creating AI-safe assignments, it’s very simple. Instead of having them write essays, I have an AI agent write an essay on this, and then they critique the output of that AI agent. Any AI agent can do that. It’s incredibly naive to think you can’t get ChatGPT to write an essay and get Claude to critique it, or even get Claude to critique the output it just had itself make. So why is it that if they can have such good judgment on all kinds of things, if they can already draft a very good contract, if they’re pretty good in the first two rounds of negotiation, we’re never going to learn to be that good in the fourth or fifth round of negotiation? What makes you so confident there are certain kinds of skills we’re not going to acquire?
Kaplan: Well, you’re kind of poking at this, if I may say, in a little bit of the wrong direction. A lot of those things they are going to be able to do, and the point is we should use them to do that, if they’re effective and safe to use. These are tools. That’s a whole other hour we could spend on. But the key skill for humans in the future is managing AI agents. That’s where we’re heading, and that’s a very important skill—it’s already an important skill. I’m watching my kids as they learn this; they become very valuable to their organizations because they’re learning to be managers. The future of white-collar work is managing these things. And if you’re good at it and you can get a great result, that’s a good thing—that’s the key skill.
But there’s lots of stuff we don’t want to use these things for—there’s just no motivation to. You’re big on the abundance agenda—we’re going to have a lot more resources in the future, at least that’s one of the assumptions in all that. What are people going to do with all this extra money? Well, they’re going to do things to have fun, to play. And the truth is, in the future, the result of this technology is that it’s going to make human-to-human contact more valuable. It’s going to make things where you’ve invested your time and effort more valuable, not less. You’re not going to want to hire a robot to give you a tour. People with enough money, on a wine tour, are going to hire a human expert. You’re not going to want to see robots playing in the Super Bowl—although that would be kind of fun—you want to see human beings doing what they do best. So the future, the funny thing, is it’s freeing us to be more human. Much of the work we’re going to be doing in the future, the way we’re going to be using these things, is to automate the work they’re capable of doing and free us up to be more human, to have more connection to other people.
Mounk: I guess I would be more confident of this if I didn’t feel that they are already surprisingly good, and likely soon will be astonishingly good, at some of the things we do think of as most human. Twenty years ago, if you’d asked—you know the beautiful question written above the philosophy department at Harvard, William James Hall; some people are sad it’s there because it’s a Bible quote, but I think it’s a lovely Bible quote—”What is man, that thou art mindful of him?”
A lot of the answer we may have given is: man is capable, and woman is capable, of writing poems, of writing symphonies, of creating beautiful works of art. And AIs at this point are pretty close to being able to do that. They certainly can write very good poems. There’s been a real leap of quality in the music they produce in the last few months. I think there’s no real reason to think it’s not going to be able to produce something that, to somebody who’s listened to five or six of Beethoven’s symphonies, wouldn’t sound plausibly like his sixth or seventh symphony. This is one of the areas in which some of the core creative pursuits of humans have now been outsourced.
I think about this with my own writing. I love writing, I love the process of formulating ideas and making them as clear as I can, because it’s a necessary act of communication, and I get into flow and enjoy the process of doing that. I’ll always write my own text because that’s something I’m proud of and that feels core to me, to being a writer. But a lot of the beauty of that is going to be lost if I know I could just prompt Claude with a few lines and it could do it just as well as me, or perhaps better. So I do worry that some of the things we call being human may, at the very least, be displaced, and perhaps in certain ways lost.
Kaplan: Well, look, there’s a lot to be said about this. Let me take you back to—I think it was, coincidentally, a 1906 essay by John Philip Sousa, who was at the time a very famous musician, the guy who wrote Stars and Stripes Forever, his marching band music. Right about that time is when the first recorded music was possible. He wrote a piece called The Menace of Mechanical Music and it’s worth reading this diatribe he wrote against recorded music, on exactly the point you just talked about, because you can see his point of view and understand it today.
What he said was: what is this going to do to the dance orchestra? This is the terrible thing—it is the joy of the nightingale’s song to express its own inner feelings, that’s what music is about, and music isn’t something that can be repeated day after day by some mechanical device. That’s not music—that was his argument, that it wasn’t music.
Now, today you would say that is music—you just did say that was music. They can write beautifully, maybe they can create beautiful music and all that—it’s derivative of everything that has come before, as arguably humans are doing as well. But the important point is it isn’t being sparked by some internal emotional state that they’re trying to communicate to you. Because that’s what we mean when we say, I read a book and I really felt what the author was trying to communicate. It’s a means of communication between human beings, for their emotional state, and it’s that simpatico that occurs—that’s what great art is about. And no, they can’t do that, Yascha, because they don’t have those feelings. It’s artificial, it’s fake. I can sit here and have a machine all day long going, I love you, I love you, I love you. Does that make me feel better? No. Why? I don’t think it feels love at all. It’s just blowing smoke.
You’ve got to distinguish between these two things. Let me give you another point that may be important. A lot of people are craftsmen who build fine furniture, for example. What happened to that skill, and our appreciation of it, when it became possible to manufacture incredibly well-made furniture at very cheap prices, and you go down to IKEA and Target and buy them right there? The answer is that skill became more valuable, not less. That’s exactly the difference. When you write for your magazine, which I have done, and I highly recommend it to everybody listening to this podcast, I think you have to sign something saying you wrote it yourself—you can use AI to help, but you have to write it yourself. So that’s my point: you don’t need to worry about this, Yascha. It’s going to play out the same way it played out with recorded music. It’s going to play out the same way it happened when photography was invented—I could go through that example from the 1860s, fascinating, all the artists were up in arms, this is not art. Now we think photographers are artists. The equivalent of the photographer today is somebody who can manage these machines to produce something of great beauty. That’s going to be the key skill.
In the rest of this conversation, Yascha and Jerry discuss what could happen when artificial intelligence becomes smarter than us, what the Etsyfication of human creativity means for writers, and whether these tools are just our slaves. This part of the conversation is reserved for paying subscribers…
Listen to this episode with a 7-day free trial
Subscribe to Yascha Mounk to listen to this post and get 7 days of free access to the full post archives.
