42m⁝ Interviews AI related

42m1⁝ We are near the end of the exponential

https://www.youtube.com/watch?v=n1E9IZfvGMA

[00:00:00] We talked three years ago. In your view, what has been the biggest update over the last three years? [00:00:05] What has been the biggest difference between what it felt like then versus now? [00:00:10] Broadly speaking, the exponential of the underlying technology has gone about as [00:00:18] I expected it to go. There’s plus or minus [00:00:23] a year or two here and there. I don’t know that I would’ve [00:00:27] predicted the specific direction of code. But when I look at the exponential, [00:00:34] it is roughly what I expected in terms of the march of the models from smart high [00:00:39] school student to smart college student to beginning to do PhD and professional stuff, [00:00:44] and in the case of code reaching beyond that. The frontier is a little bit uneven, but it’s [00:00:49] roughly what I expected. What has been the most surprising [00:00:55] thing is the lack of public recognition of how close we are to the end of the exponential. [00:01:02] To me, it is absolutely wild that you have people — within the bubble and outside the [00:01:09] bubble — talking about the same tired, old hot-button political issues, when we are [00:01:19] near the end of the exponential. I want to understand what that [00:01:24] exponential looks like right now. The first question I asked you when [00:01:27] we recorded three years ago was, “what’s up with scaling and why does it work?” [00:01:31] I have a similar question now, but it feels more complicated. [00:01:35] At least from the public’s point of view, three years ago there were well-known public trends [00:01:41] across many orders of magnitude of compute where you could see how the loss improves. [00:01:45] Now we have RL scaling and there’s no publicly known scaling law for it. [00:01:49] It’s not even clear what the story is. Is this supposed to be teaching the model skills? [00:01:54] Is it supposed to be teaching meta-learning? What is the scaling hypothesis at this point? [00:01:59] I actually have the same hypothesis I had even all the way back in 2017. [00:02:06] I think I talked about it last time, but I wrote a doc called “The Big Blob of Compute Hypothesis”. [00:02:12] It wasn’t about the scaling of language models in particular. [00:02:15] When I wrote it GPT-1 had just come out. That was one among many things. [00:02:22] Back in those days there was robotics. People tried to work on reasoning as [00:02:26] a separate thing from language models, and there was scaling of the kind of RL [00:02:30] that happened in AlphaGo and in Dota at OpenAI. People remember StarCraft at DeepMind, AlphaStar. [00:02:43] It was written as a more general document. Rich Sutton put out “The Bitter [00:02:52] Lesson” a couple years later. The hypothesis is basically the same. [00:02:57] What it says is that all the cleverness, all the techniques, all the “we need a new method to do [00:03:04] something”, that doesn’t matter very much. There are only a few things that matter. [00:03:08] I think I listed seven of them. One is how much raw compute you have. [00:03:13] The second is the quantity of data. The third is the quality and distribution of data. [00:03:20] It needs to be a broad distribution. The fourth is how long you train for. [00:03:27] The fifth is that you need an objective function that can scale to the moon. [00:03:32] The pre-training objective function is one such objective function. [00:03:36] Another is the RL objective function that says you have a goal, [00:03:42] you’re going to go out and reach the goal. Within that, there’s objective rewards like [00:03:48] you see in math and coding, and there’s more subjective rewards like you see in [00:03:52] RLHF or higher-order versions of that. Then the sixth and seventh were things [00:03:59] around normalization or conditioning, just getting the numerical stability [00:04:04] so that the big blob of compute flows in this laminar way instead of running into problems. [00:04:11] That was the hypothesis, and it’s a hypothesis I still hold. [00:04:15] I don’t think I’ve seen very much that is not in line with it. [00:04:21] The pre-training scaling laws were one example of what we see there. Those have continued going. [00:04:31] Now it’s been widely reported, we feel good about pre-training. [00:04:35] It’s continuing to give us gains. What has changed is that now we’re [00:04:41] also seeing the same thing for RL. We’re seeing a pre-training phase [00:04:46] and then an RL phase on top of that. With RL, it’s actually just the same. [00:04:55] Even other companies have published things in some of their releases that say, “We train the [00:05:05] model on math contests — AIME or other things — and how well the model does is log-linear in [00:05:14] how long we’ve trained it.” We see that as well, [00:05:17] and it’s not just math contests. It’s a wide variety of RL tasks. [00:05:21] We’re seeing the same scaling in RL that we saw for pre-training. [00:05:27] You mentioned Rich Sutton and “The Bitter Lesson”. I interviewed him last year, [00:05:31] and he’s actually very non-LLM-pilled. I don’t know if this is his perspective, [00:05:38] but one way to paraphrase his objection is: Something which possesses the true core of human [00:05:44] learning would not require all these billions of dollars of data and compute and these bespoke [00:05:51] environments, to learn how to use Excel, how to use PowerPoint, how to navigate a web browser. [00:05:57] The fact that we have to build in these skills using these RL environments hints that we are [00:06:04] actually lacking a core human learning algorithm. So we’re scaling the wrong thing. That does raise [00:06:13] the question. Why are we doing all this RL scaling if we think there’s something that’s going to be [00:06:16] human-like in its ability to learn on the fly? I think this puts together several things that [00:06:23] should be thought of differently. There is a genuine puzzle here, [00:06:29] but it may not matter. In fact, I would guess it probably [00:06:33] doesn’t matter. There is an interesting thing. Let me take the RL out of it for a second, because I [00:06:37] actually think it’s a red herring to say that RL is any different from pre-training in this matter. [00:06:43] If we look at pre-training scaling, it was very interesting [00:06:47] back in 2017 when Alec Radford was doing GPT-1. The models before GPT-1 were trained on datasets [00:06:59] that didn’t represent a wide distribution of text. You had very standard language [00:07:08] modeling benchmarks. GPT-1 itself was trained on [00:07:11] a bunch of fanfiction, I think actually. It was literary text, which is a very [00:07:17] small fraction of the text you can get. In those days it was like a billion words [00:07:23] or something, so small datasets representing a pretty narrow distribution of what you can [00:07:32] see in the world. It didn’t generalize well. If you did better on some fanfiction corpus, [00:07:43] it wouldn’t generalize that well to other tasks. We had all these measures. We had [00:07:47] all these measures of how well it did at predicting all these other kinds of texts. [00:07:55] It was only when you trained over all the tasks on the internet — when you did a general internet [00:08:01] scrape from something like Common Crawl or scraping links in Reddit, which is what we did for [00:08:06] GPT-2 — that you started to get generalization. I think we’re seeing the same thing on RL. [00:08:15] We’re starting first with simple RL tasks like training on math competitions, then moving to [00:08:24] broader training that involves things like code. Now we’re moving to many other tasks. [00:08:31] I think then we’re going to increasingly get generalization. [00:08:35] So that kind of takes out the RL vs. pre-training side of it. [00:08:39] But there is a puzzle either way, which is that in pre-training we use trillions of tokens. [00:08:50] Humans don’t see trillions of words. So there is an actual sample [00:08:54] efficiency difference here. There is actually something different here. [00:08:59] The models start from scratch and they need much more training. [00:09:06] But we also see that once they’re trained, if we give them a long context length of [00:09:15] a million — the only thing blocking long context is inference — they’re very good at [00:09:17] learning and adapting within that context. So I don’t know the full answer to this. [00:09:24] I think there’s something going on where pre-training is not like [00:09:28] the process of humans learning, but it’s somewhere between the process of humans [00:09:32] learning and the process of human evolution. We get many of our priors from evolution. [00:09:38] Our brain isn’t just a blank slate. Whole books have been written about this. [00:09:43] The language models are much more like blank slates. [00:09:45] They literally start as random weights, whereas the human brain starts with all these regions [00:09:50] connected to all these inputs and outputs. Maybe we should think of pre-training — and [00:09:56] for that matter, RL as well — as something that exists in the middle space between [00:10:02] human evolution and human on-the-spot learning. And we should think of the in-context learning [00:10:10] that the models do as something between long-term human learning and short-term human learning. [00:10:17] So there’s this hierarchy. There’s evolution, there’s long-term learning, there’s short-term [00:10:22] learning, and there’s just human reaction. The LLM phases exist along this spectrum, [00:10:28] but not necessarily at exactly the same points. There’s no analog to some of the human modes [00:10:34] of learning the LLMs are falling in between the points. Does that make sense? [00:10:40] Yes, although some things are still a bit confusing. [00:10:42] For example, if the analogy is that this is like evolution so it’s fine that it’s [00:10:45] not sample efficient, then if we’re going to get super sample-efficient [00:10:51] agent from in-context learning, why are we bothering to build all these RL environments? [00:10:56] There are companies whose work seems to be teaching models how to use this API, [00:11:00] how to use Slack, how to use whatever. It’s confusing to me why there’s so much emphasis [00:11:04] on that if the kind of agent that can just learn on the fly is emerging or has already emerged. [00:11:11] I can’t speak for the emphasis of anyone else. I can only talk about how we think about it. [00:11:20] The goal is not to teach the model every possible skill within RL, [00:11:25] just as we don’t do that within pre-training. Within pre-training, we’re not trying to expose [00:11:29] the model to every possible way that words could be put together. [00:11:37] Rather, the model trains on a lot of things and then reaches generalization across pre-training. [00:11:43] That was the transition from GPT-1 to GPT-2 that I saw up close. The model reaches a point. I had [00:11:53] these moments where I was like, “Oh yeah, you just give the model a list of numbers — this is [00:12:01] the cost of the house, this is the square feet of the house — and the model completes the pattern [00:12:05] and does linear regression.” Not great, but it does it, [00:12:08] and it’s never seen that exact thing before. So to the extent that we are building these [00:12:16] RL environments, the goal is very similar to what was done five or ten years ago with pre-training. [00:12:26] We’re trying to get a whole bunch of data, not because we want to cover a specific document or a [00:12:32] specific skill, but because we want to generalize. I think the framework you’re laying down obviously [00:12:39] makes sense. We’re making progress toward AGI. Nobody at this point disagrees we’re going to [00:12:46] achieve AGI this century. The crux is you say we’re [00:12:49] hitting the end of the exponential. Somebody else looks at this and says, [00:12:55] “We’ve been making progress since 2012, and by 2035 we’ll have a human-like agent.” [00:13:04] Obviously we’re seeing in these models the kinds of things that evolution did, [00:13:07] or that learning within a human lifetime does. I want to understand what you’re seeing [00:13:11] that makes you think it’s one year away and not ten years away. [00:13:17] There are two claims you could make here, one stronger and one weaker. [00:13:26] Starting with the weaker claim, when I first saw the scaling back in 2019, [00:13:35] I wasn’t sure. This was a 50/50 thing. I thought I saw something. My claim was that this [00:13:43] was much more likely than anyone thinks. Maybe there’s a 50% chance this happens. [00:13:51] On the basic hypothesis of, as you put it, within ten years we’ll get to what I call a “country of [00:14:00] geniuses in a data center”, I’m at 90% on that. It’s hard to go much higher than 90% [00:14:06] because the world is so unpredictable. Maybe the irreducible uncertainty puts us at 95%, [00:14:13] where you get to things like multiple companies having internal turmoil, Taiwan gets invaded, [00:14:24] all the fabs get blown up by missiles. Now you’ve jinxed us, Dario. [00:14:30] You could construct a 5% world where things get delayed for ten years. [00:14:43] There’s another 5% which is that I’m very confident on tasks that can be verified. [00:14:50] With coding, except for that irreducible uncertainty, [00:14:54] I think we’ll be there in one or two years. There’s no way we will not be there in ten years [00:14:58] in terms of being able to do end-to-end coding. My one little bit of fundamental uncertainty, [00:15:05] even on long timescales, is about tasks that aren’t verifiable: planning a mission to Mars; [00:15:14] doing some fundamental scientific discovery like CRISPR; writing a novel. [00:15:21] It’s hard to verify those tasks. I am almost certain we have a [00:15:27] reliable path to get there, but if there’s a little bit of uncertainty it’s there. [00:15:34] On the ten-year timeline I’m at 90%, which is about as certain as you can be. [00:15:40] I think it’s crazy to say that this won’t happen by 2035. [00:15:46] In some sane world, it would be outside the mainstream. [00:15:48] But the emphasis on verification hints to me a lack of belief that these models are generalized. [00:15:58] If you think about humans, we’re both good at things for which we get verifiable reward [00:16:03] and things for which we don’t. No, this is why I’m almost sure. [00:16:07] We already see substantial generalization from things that verify to things that [00:16:12] don’t. We’re already seeing that. But it seems like you were emphasizing [00:16:15] this as a spectrum which will split apart which domains in which we see more progress. [00:16:21] That doesn’t seem like how humans get better. The world in which we don’t get there is the world [00:16:27] in which we do all the verifiable things. Many of them generalize, [00:16:34] but we don’t fully get there. We don’t fully color in the other side [00:16:40] of the box. It’s not a binary thing. Even if generalization is weak and you can only do [00:16:47] verifiable domains, it’s not clear to me you could automate software engineering in such a world. [00:16:49] You are “a software engineer” in some sense, but part of being a software engineer for you involves [00:16:58] writing long memos about your grand vision. I don’t think that’s part of the job of SWE. [00:17:03] That’s part of the job of the company, not SWE specifically. [00:17:04] But SWE does involve design documents and other things like that. [00:17:10] The models are already pretty good at writing comments. [00:17:14] Again, I’m making much weaker claims here than I believe, to distinguish between two things. [00:17:24] We’re already almost there for software engineering. [00:17:28] By what metric? There’s one metric which is how many lines of code are written by AI. [00:17:32] If you consider other productivity improvements in the history of software engineering, [00:17:36] compilers write all the lines of software. There’s a difference between how many lines [00:17:40] are written and how big the productivity improvement is. “We’re almost there” meaning… [00:17:47] How big is the productivity improvement, not just how many lines are written by AI? [00:17:52] I actually agree with you on this. I’ve made a series of predictions on [00:17:57] code and software engineering. I think people have repeatedly misunderstood them. [00:18:03] Let me lay out the spectrum. About eight or nine months ago, [00:18:09] I said the AI model will be writing 90% of the lines of code in three to six months. [00:18:16] That happened, at least at some places. It happened at Anthropic, happened with [00:18:21] many people downstream using our models. But that’s actually a very weak criterion. [00:18:27] People thought I was saying that we won’t need 90% of the software engineers. Those things are worlds [00:18:32] apart. The spectrum is: 90% of code is written by the model, 100% of code is written by the model. [00:18:41] That’s a big difference in productivity. 90% of the end-to-end SWE tasks — including [00:18:47] things like compiling, setting up clusters and environments, testing features, [00:18:54] writing memos — are done by the models. 100% of today’s SWE tasks are done by the models. [00:19:02] Even when that happens, it doesn’t mean software engineers are out of a job. [00:19:06] There are new higher-level things they can do, where they can manage. [00:19:10] Then further down the spectrum, there’s 90% less demand for SWEs, which I think [00:19:15] will happen but this is a spectrum. I wrote about it in “The Adolescence [00:19:21] of Technology” where I went through this kind of spectrum with farming. [00:19:26] I actually totally agree with you on that. [00:19:29] These are very different benchmarks from each other, [00:19:32] but we’re proceeding through them super fast. Part of your vision is that going from 90 to 100 [00:19:38] is going to happen fast, and that it leads to huge productivity improvements. [00:19:45] But what I notice is that even in greenfield projects people start with Claude Code or [00:19:49] something, people report starting a lot of projects… Do we see in the world out there [00:19:54] a renaissance of software, all these new features that wouldn’t exist otherwise? [00:19:58] At least so far, it doesn’t seem like we see that. So that does make me wonder. [00:20:02] Even if I never had to intervene with Claude Code, the world is complicated. [00:20:09] Jobs are complicated. Closing the loop on self-contained systems, whether it’s just [00:20:14] writing software or something, how much broader gains would we see just from that? [00:20:20] Maybe that should dilute our estimation of the “country of geniuses”. [00:20:24] I simultaneously agree with you that it’s a reason why these things don’t happen instantly, [00:20:35] but at the same time, I think the effect is gonna be very fast. [00:20:41] You could have these two poles. One is that AI is not going to make [00:20:47] progress. It’s slow. It’s going to take forever to diffuse within the economy. [00:20:52] Economic diffusion has become one of these buzzwords that’s a reason why [00:20:56] we’re not going to make AI progress, or why AI progress doesn’t matter. [00:21:00] The other axis is that we’ll get recursive self-improvement, the whole thing. [00:21:05] Can’t you just draw an exponential line on the curve? [00:21:08] We’re going to have Dyson spheres around the sun so many nanoseconds after we get recursive. [00:21:17] I’m completely caricaturing the view here, but there are these two extremes. [00:21:23] But what we’ve seen from the beginning, at least if you look within Anthropic, there’s this bizarre [00:21:30] 10x per year growth in revenue that we’ve seen. So in 2023, it was zero to $100million.[00:21:38]In2024,itwas$100 million to $1billion.In2025,itwas$1 billion to $9-10billion.[00:21:46]Youguysshouldhavejustboughtabilliondollarsofyourownproductssoyoucouldjust\dots [00:21:50]Andthefirstmonthofthisyear,thatexponentialis...[00:21:54]Youwouldthinkitwouldslowdown,butweaddedanotherfewbilliontorevenueinJanuary.[00:22:05]Obviouslythatcurveca{n}^{\prime }tgoonforever.TheGDPisonlysolarge.[00:22:10]Iwouldevenguessthatitbendssomewhatthisyear,butthatisafastcurve.Tha{t}^{\prime }sareally[00:22:20]fastcurve.Iwouldbetitstaysprettyfastevenasthescalegoestotheentireeconomy.[00:22:25]SoIthinkweshouldbethinkingaboutthismiddleworldwherethingsareextremelyfast,butnot[00:22:34]instant,wheretheytaketimebecauseofeconomicdiffusion,becauseoftheneedtoclosetheloop.[00:22:39]Becausei{t}^{\prime }sfiddly:"Ihavetodochangemanagementwithinmyenterprise\dots Isetthisup,[00:22:50]butIhavetochangethesecuritypermissionsonthisinordertomakeitactuallywork\dots [00:22:55]Ihadthisoldpieceofsoftwarethatchecksthemodelbeforei{t}^{\prime }scompiled[00:23:01]andreleasedandIhavetorewriteit.Yes,themodelcandothat,butIhave[00:23:05]totellthemodeltodothat.Ithastotaketimetodothat."[00:23:10]SoIthinkeverythingw{e}^{\prime }veseensofariscompatiblewiththeideathatther{e}^{\prime }sonefast[00:23:17]exponentialtha{t}^{\prime }sthecapabilityofthemodel.Thenther{e}^{\prime }sanotherfastexponential[00:23:22]tha{t}^{\prime }sdownstreamofthat,whichisthediffusionofthemodelintotheeconomy.[00:23:26]Notinstant,notslow,muchfasterthananyprevioustechnology,butithasitslimits.[00:23:37]WhenIlookinsideAnthropic,whenIlookatourcustomers:fastadoption,butnotinfinitelyfast.[00:23:44]CanItryahottakeonyou?Yeah.[00:23:45]Ifeellikediffusioniscopethatpeoplesay.Whenthemodelis{n}^{\prime }tabletodosomething,[00:23:51]the{y}^{\prime }relike,"oh,buti{t}^{\prime }sadiffusionissue."Butthenyoushouldusethecomparisontohumans.[00:23:56]YouwouldthinkthattheinherentadvantagesthatAIshavewouldmakediffusionamucheasier[00:24:01]problemfornewAIsgettingonboardedthannewhumansgettingonboarded.[00:24:06]AnAIcanreadyourentireSlackandyourdriveinminutes.[00:24:08]Theycansharealltheknowledgethattheothercopiesofthesameinstancehave.[00:24:12]Youdo{n}^{\prime }thavethisadverseselectionproblemwhenyo{u}^{\prime }rehiringAI,soyou[00:24:14]canjusthirecopiesofavettedAImodel.Hiringahumanissomuchmoreofahassle.[00:24:20]Peoplehirehumansallthetime.Wepayhumansupwardsof$50 trillion [00:24:23] in wages because they’re useful, even though in principle it would be much easier to integrate [00:24:29] AIs into the economy than it is to hire humans. The diffusion doesn’t really explain. [00:24:34] I think diffusion is very real and doesn’t exclusively have [00:24:41] to do with limitations on the AI models. Again, there are people who use diffusion [00:24:49] as kind of a buzzword to say this isn’t a big deal. I’m not talking about that. I’m [00:24:54] not talking about how AI will diffuse at the speed of previous technologies. [00:24:58] I think AI will diffuse much faster than previous technologies have, but not infinitely fast. [00:25:04] I’ll just give an example of this. There’s Claude Code. Claude Code is extremely easy to set up. [00:25:10] If you’re a developer, you can just start using Claude Code. [00:25:14] There is no reason why a developer at a large enterprise should not be adopting [00:25:19] Claude Code as quickly as an individual developer or developer at a startup. [00:25:25] We do everything we can to promote it. We sell Claude Code to enterprises. [00:25:31] Big enterprises, big financial companies, big pharmaceutical companies, all of them are adopting [00:25:38] Claude Code much faster than enterprises typically adopt new technology. But again, [00:25:46] it takes time. Any given feature or any given product, like Claude Code or Cowork, will get [00:25:54] adopted by the individual developers who are on Twitter all the time, by the Series A startups, [00:26:02] many months faster than they will get adopted by a large enterprise that does food sales. [00:26:11] There are just a number of factors. You have to go through legal, [00:26:14] you have to provision it for everyone. It has to pass security and compliance. [00:26:20] The leaders of the company who are further away from the AI revolution are forward-looking, [00:26:26] but they have to say, “Oh, it makes sense for us to spend 50 million. [00:26:31] This is what this Claude Code thing is. This is why it helps our company. [00:26:35] This is why it makes us more productive.” Then they have to explain [00:26:37] to the people two levels below. They have to say, “Okay, we have 3,000 developers. [00:26:42] Here’s how we’re going to roll it out to our developers.” [00:26:45] We have conversations like this every day. We are doing everything we can to make [00:26:50] Anthropic’s revenue grow 20 or 30x a year instead of 10x a year. [00:26:57] Again, many enterprises are just saying, “This is so productive. [00:27:02] We’re going to take shortcuts in our usual procurement process.” [00:27:05] They’re moving much faster than when we tried to sell them just [00:27:08] the ordinary API, which many of them use. Claude Code is a more compelling product, [00:27:13] but it’s not an infinitely compelling product. I don’t think even AGI or powerful AI or [00:27:19] “country of geniuses in a data center” will be an infinitely compelling product. [00:27:22] It will be a compelling product enough maybe to get 3-5x, or 10x, a year of growth, even when [00:27:28] you’re in the hundreds of billions of dollars, which is extremely hard to do and has never been [00:27:32] done in history before, but not infinitely fast. I buy that it would be a slight slowdown. [00:27:36] Maybe this is not your claim, but sometimes people talk about this like, [00:27:39] “Oh, the capabilities are there, but because of diffusion… otherwise we’re basically at AGI”. [00:27:46] I don’t believe we’re basically at AGI. I think if you had the “country [00:27:49] of geniuses in a data center”… If we had the “country of geniuses [00:27:53] in a data center”, we would know it. We would know it if you had the [00:27:57] “country of geniuses in a data center”. Everyone in this room would know it. [00:28:01] Everyone in Washington would know it. People in rural parts might not know it, [00:28:07] but we would know it. We don’t have that now. That is very clear. [00:29:42] Coming back to concrete prediction… Because there are so many different things to disambiguate, [00:29:47] it can be easy to talk past each other when we’re talking about capabilities. [00:29:50] For example, when I interviewed you three years ago, I asked you a prediction about what [00:29:54] we should expect three years from now. You were right. You said, “We should expect systems which, [00:30:00] if you talk to them for the course of an hour, it’s hard to tell them apart from [00:30:04] a generally well-educated human.” I think you were right about that. [00:30:07] I think spiritually I feel unsatisfied because my internal expectation was that such a system could [00:30:13] automate large parts of white-collar work. So it might be more productive to talk about [00:30:17] the actual end capabilities you want from such a system. [00:30:21] I will basically tell you where I think we are. Let me ask a very specific question so that [00:30:28] we can figure out exactly what kinds of capabilities we should think about soon. [00:30:32] Maybe I’ll ask about it in the context of a job I understand well, not because it’s the most [00:30:36] relevant job, but just because I can evaluate the claims about it. Take video editors. I have [00:30:42] video editors. Part of their job involves learning about our audience’s preferences, [00:30:47] learning about my preferences and tastes, and the different trade-offs we have. [00:30:50] They’re, over the course of many months, building up this understanding of context. [00:30:55] The skill and ability they have six months into the job, a model that can [00:30:58] pick up that skill on the job on the fly, when should we expect such an AI system? [00:31:04] I guess what you’re talking about is that we’re doing this interview for three hours. [00:31:09] Someone’s going to come in, someone’s going to edit it. [00:31:11] They’re going to be like, “Oh, I don’t know, Dario scratched his head and we could edit that out.” [00:31:19] “Magnify that.” “There was this long [00:31:22] discussion that is less interesting to people. There’s another thing that’s more interesting [00:31:27] to people, so let’s make this edit.” I think the “country of geniuses in [00:31:33] a data center” will be able to do that. The way it will be able to do that is it will [00:31:38] have general control of a computer screen. You’ll be able to feed this in. [00:31:43] It’ll be able to also use the computer screen to go on the web, look at all your previous [00:31:49] interviews, look at what people are saying on Twitter in response to your interviews, [00:31:54] talk to you, ask you questions, talk to your staff, look at the history of edits [00:31:59] that you did, and from that, do the job. I think that’s dependent on several things. [00:32:06] I think this is one of the things that’s actually blocking deployment: [00:32:10] getting to the point on computer use where the models are really masters at using the computer. [00:32:16] We’ve seen this climb in benchmarks, and benchmarks are always imperfect measures. [00:32:20] But I think when we first released computer use a year and a quarter ago, OSWorld was at maybe 15%. [00:32:33] I don’t remember exactly, but we’ve climbed from that to 65-70%. [00:32:40] There may be harder measures as well, but I think computer use has to pass a point of reliability. [00:32:46] Can I just follow up on that before you move on to the next point? [00:32:50] For years, I’ve been trying to build different internal LLM tools for myself. [00:32:54] Often I have these text-in, text-out tasks, which should be dead center [00:32:59] in the repertoire of these models. Yet I still hire humans to do them. [00:33:03] If it’s something like, “identify what the best clips would be in this transcript”, [00:33:07] maybe the LLMs do a seven-out-of-ten job on them. But there’s not this ongoing way I can engage [00:33:12] with them to help them get better at the job the way I could with a human employee. [00:33:16] That missing ability, even if you solve computer use, would still block [00:33:20] my ability to offload an actual job to them. This gets back to what we were talking about [00:33:28] before with learning on the job. It’s very interesting. I think with the coding agents, [00:33:34] I don’t think people would say that learning on the job is what is preventing the coding agents [00:33:39] from doing everything end to end. They keep getting better. We have engineers [00:33:46] at Anthropic who don’t write any code. When I look at the productivity, to your [00:33:51] previous question, we have folks who say, “This GPU kernel, this chip, I used to write it myself. [00:33:58] I just have Claude do it.” There’s this enormous improvement in productivity. [00:34:04] When I see Claude Code, familiarity with the codebase or a feeling that the model [00:34:13] hasn’t worked at the company for a year, that’s not high up on the list of complaints I see. [00:34:18] I think what I’m saying is that we’re kind of taking a different path. [00:34:22] Don’t you think with coding that’s because there [00:34:24] is an external scaffold of memory which exists instantiated in the codebase? [00:34:28] I don’t know how many other jobs have that. Coding made fast progress precisely because [00:34:33] it has this unique advantage that other economic activity doesn’t. [00:34:37] But when you say that, what you’re implying is that by reading the codebase into the context, [00:34:44] I have everything that the human needed to learn on the job. [00:34:48] So that would be an example of—whether it’s written or not, whether it’s available or [00:34:54] not—a case where everything you needed to know you got from the context window. [00:35:00] What we think of as learning—“I started this job, it’s going to take me six months to understand the [00:35:05] code base”—the model just did it in the context. I honestly don’t know how to think about [00:35:09] this because there are people who qualitatively report what you’re saying. [00:35:16] I’m sure you saw last year, there was a major study where they had experienced developers try [00:35:21] to close pull requests in repositories that they were familiar with. Those developers reported an [00:35:28] uplift. They reported that they felt more productive with the use of these models. [00:35:31] But in fact, if you look at their output and how much was actually merged back in, [00:35:35] there was a 20% downlift. They were less productive [00:35:37] as a result of using these models. So I’m trying to square the qualitative [00:35:40] feeling that people feel with these models versus, 1) in a macro level, [00:35:44] where is this renaissance of software? And then 2) when people do these independent [00:35:48] evaluations, why are we not seeing the productivity benefits we would expect? [00:35:53] Within Anthropic, this is just really unambiguous. We’re under an incredible amount of commercial [00:35:59] pressure and make it even harder for ourselves because we have all this safety stuff we do that [00:36:03] I think we do more than other companies. The pressure to survive economically [00:36:11] while also keeping our values is just incredible. We’re trying to keep this 10x revenue curve going. [00:36:18] There is zero time for bullshit. There is zero time for feeling [00:36:23] like we’re productive when we’re not. These tools make us a lot more productive. [00:36:30] Why do you think we’re concerned about competitors using the tools? [00:36:34] Because we think we’re ahead of the competitors. We wouldn’t be going through all this trouble if [00:36:43] this were secretly reducing our productivity. We see the end productivity every few [00:36:49] months in the form of model launches. There’s no kidding yourself about this. [00:36:54] The models make you more productive. 1) People feeling like they’re productive is [00:37:00] qualitatively predicted by studies like this. But 2) if I just look at the end output, [00:37:04] obviously you guys are making fast progress. But the idea was supposed to be that with [00:37:10] recursive self-improvement, you make a better AI, the AI helps you build a [00:37:14] better next AI, et cetera, et cetera. What I see instead—if I look at you, [00:37:18] OpenAI, DeepMind—is that people are just shifting around the podium every few months. [00:37:22] Maybe you think that stops because you’ve won or whatever. [00:37:25] But why are we not seeing the person with the best coding model have this lasting [00:37:31] advantage if in fact there are these enormous productivity gains from the last coding model. [00:37:38] I think my model of the situation is that there’s an advantage that’s gradually growing. [00:37:45] I would say right now the coding models give maybe, I don’t know, [00:37:51] a 15-20% total factor speed up. That’s my view. Six months ago, it was maybe 5%. [00:38:01] So it didn’t matter. 5% doesn’t register. It’s now just getting to the point where it’s [00:38:06] one of several factors that kind of matters. That’s going to keep speeding up. [00:38:12] I think six months ago, there were several companies that were at roughly the same [00:38:18] point because this wasn’t a notable factor, but I think it’s starting to speed up more and more. [00:38:25] I would also say there are multiple companies that write models that are used for code and we’re not [00:38:32] perfectly good at preventing some of these other companies from using our models internally. [00:38:41] So I think everything we’re seeing is consistent with this kind of snowball model. [00:38:52] Again, my theme in all of this is all of this is soft takeoff, soft, smooth exponentials, [00:39:00] although the exponentials are relatively steep. So we’re seeing this snowball gather momentum [00:39:05] where it’s like 10%, 20%, 25%, 40%. As you go, Amdahl’s law, you have [00:39:13] to get all the things that are preventing you from closing the loop out of the way. [00:39:17] But this is one of the biggest priorities within Anthropic. [00:39:22] Stepping back, before in the stack we were talking about when do we get this on-the-job learning? [00:39:29] It seems like the point you were making on the coding thing is that we actually [00:39:32] don’t need on-the-job learning. You can have tremendous productivity [00:39:36] improvements, you can have potentially trillions of dollars of revenue for AI companies, without [00:39:40] this basic human ability to learn on the job. Maybe that’s not your claim, you should clarify. [00:39:47] But in most domains of economic activity, people say, “I hired somebody, they weren’t that useful [00:39:53] for the first few months, and then over time they built up the context, understanding.” [00:39:58] It’s actually hard to define what we’re talking about here. [00:40:00] But they got something and then now they’re a powerhorse and they’re so valuable to us. [00:40:05] If AI doesn’t develop this ability to learn on the fly, I’m a bit skeptical that we’re going to see [00:40:12] huge changes to the world without that ability. I think two things here. There’s the state [00:40:17] of the technology right now. Again, we have these two stages. [00:40:22] We have the pre-training and RL stage where you throw a bunch of data and tasks into [00:40:27] the models and then they generalize. So it’s like learning, but it’s like [00:40:31] learning from more data and not learning over one human or one model’s lifetime. [00:40:38] So again, this is situated between evolution and human learning. [00:40:42] But once you learn all those skills, you have them. [00:40:45] Just like with pre-training, just how the models know more, if I look at a pre-trained model, [00:40:52] it knows more about the history of samurai in Japan than I do. [00:40:55] It knows more about baseball than I do. It knows more about low-pass filters [00:41:03] and electronics, all of these things. Its knowledge is way broader than mine. [00:41:08] So I think even just that may get us to the point where the models are better at everything. [00:41:18] We also have, again, just with scaling the kind of existing setup, the in-context learning. [00:41:24] I would describe it as kind of like human on-the-job learning, [00:41:27] but a little weaker and a little short term. You look at in-context learning and if you give [00:41:33] the model a bunch of examples it does get it. There’s real learning that happens in context. [00:41:38] A million tokens is a lot. That can be days of human learning. [00:41:42] If you think about the model reading a million words, how long would it [00:41:50] take me to read a million? Days or weeks at least. So you have these two things. [00:41:57] I think these two things within the existing paradigm may just be enough to get you the [00:42:01] “country of geniuses in a data center”. I don’t know for sure, but I think [00:42:04] they’re going to get you a large fraction of it. There may be gaps, but I certainly think that just [00:42:10] as things are, this is enough to generate trillions of dollars of revenue. That’s one. Two, [00:42:17] is this idea of continual learning, this idea of a single model learning on the job. [00:42:24] I think we’re working on that too. There’s a good chance that in the next [00:42:29] year or two, we also solve that. Again, I think you get most [00:42:36] of the way there without it. The trillions of dollars a year market, [00:42:45] maybe all of the national security implications and the safety implications that I wrote about in [00:42:49] “Adolescence of Technology” can happen without it. But we, and I imagine others, are working on it. [00:42:57] There’s a good chance that we will get there within the next year or two. [00:43:03] There are a bunch of ideas. I won’t go into all of them in detail, but [00:43:07] one is just to make the context longer. There’s nothing preventing [00:43:10] longer contexts from working. You just have to train at longer contexts [00:43:14] and then learn to serve them at inference. Both of those are engineering problems that [00:43:18] we are working on and I would assume others are working on them as well. [00:43:22] This context length increase, it seemed like there was a period from 2020 to 2023 [00:43:26] where from GPT-3 to GPT-4 Turbo, there was an increase from 2000 context lengths to 128K. [00:43:31] I feel like for the two-ish years since then, we’ve been in the same-ish ballpark. [00:43:37] When context lengths get much longer than that, people report qualitative [00:43:41] degradation in the ability of the model to consider that full context. [00:43:47] So I’m curious what you’re internally seeing that makes you think, “10 million contexts, [00:43:50] 100 million contexts to get six months of human learning and building context”. [00:43:54] This isn’t a research problem. This is an engineering and inference problem. [00:43:58] If you want to serve long context, you have to store your entire KV cache. [00:44:06] It’s difficult to store all the memory in the GPUs, to juggle the memory around. [00:44:11] I don’t even know the details. At this point, this is at a level of detail [00:44:15] that I’m no longer able to follow, although I knew it in the GPT-3 era. “These are the weights, [00:44:21] these are the activations you have to store…” But these days the whole thing is flipped [00:44:26] because we have MoE models and all of that. Regarding this degradation you’re talking about, [00:44:34] without getting too specific, there’s two things. There’s the context length you train at and [00:44:41] there’s a context length that you serve at. If you train at a small context length [00:44:45] and then try to serve at a long context length, maybe you get these degradations. [00:44:49] It’s better than nothing, you might still offer it, but you get these degradations. [00:44:52] Maybe it’s harder to train at a long context length. [00:44:56] I want to, at the same time, ask about maybe some rabbit holes. [00:45:01] Wouldn’t you expect that if you had to train on longer context length, [00:45:04] that would mean that you’re able to get less samples in for the same amount of compute? [00:45:10] Maybe it’s not worth diving deep on that. I want to get an answer to the [00:45:14] bigger picture question. I don’t feel a preference [00:45:20] for a human editor that’s been working for me for six months versus an AI that’s been [00:45:25] working with me for six months, what year do you predict that that will be the case? [00:45:33] My guess for that is there’s a lot of problems where basically we can do this when we have [00:45:38] the “country of geniuses in a data center”. My picture for that, if you made me guess, is [00:45:48] one to two years, maybe one to three years. It’s really hard to tell. I have a strong view—99%, [00:45:54] 95%—that all this will happen in 10 years. I think that’s just a super safe bet. [00:46:00] I have a hunch—this is more like a 50/50 thing—that it’s going to be more like [00:46:04] one to two, maybe more like one to three. So one to three years. Country of geniuses, [00:46:10] and the slightly less economically valuable task of editing videos. [00:46:14] It seems pretty economically valuable, let me tell you. [00:46:17] It’s just there are a lot of use cases like that. There are a lot of similar ones. [00:46:20] So you’re predicting that within one to three years. [00:46:23] And then, generally, Anthropic has predicted that by late ‘26 or early ‘27 we will have AI systems [00:46:28] that “have the ability to navigate interfaces available to humans doing digital work today, [00:46:34] intellectual capabilities matching or exceeding that of Nobel Prize winners, and the ability to [00:46:38] interface with the physical world”. You gave an interview two months ago [00:46:42] with DealBook where you were emphasizing your company’s more responsible compute [00:46:48] scaling as compared to your competitors. I’m trying to square these two views. [00:46:52] If you really believe that we’re going to have a country of geniuses, you want as [00:46:57] big a data center as you can get. There’s no reason to slow down. [00:47:00] The TAM of a Nobel Prize winner, that can actually do everything a Nobel Prize [00:47:04] winner can do, is trillions of dollars. So I’m trying to square this conservatism, [00:47:10] which seems rational if you have more moderate timelines, with your stated views about progress. [00:47:16] It actually all fits together. We go back to this fast, but not infinitely fast, diffusion. [00:47:23] Let’s say that we’re making progress at this rate. The technology is making progress this fast. [00:47:29] I have very high conviction that we’re going to get there within a few years. [00:47:39] I have a hunch that we’re going to get there within a year or two. [00:47:41] So there’s a little uncertainty on the technical side, but pretty strong [00:47:46] confidence that it won’t be off by much. What I’m less certain about is, again, [00:47:51] the economic diffusion side. I really do believe that we could [00:47:56] have models that are a country of geniuses in the data center in one to two years. [00:48:03] One question is: How many years after that do the trillions in revenue start rolling in? [00:48:14] I don’t think it’s guaranteed that it’s going to be immediate. [00:48:19] It could be one year, it could be two years, I could even stretch it to five [00:48:27] years although I’m skeptical of that. So we have this uncertainty. Even if the technology goes as [00:48:35] fast as I suspect that it will, we don’t know exactly how fast it’s going to drive revenue. [00:48:41] We know it’s coming, but with the way you buy these data centers, if you’re off by a couple [00:48:47] years, that can be ruinous. It is just like how I [00:48:50] wrote in “Machines of Loving Grace”. I said I think we might get this powerful AI, [00:48:55] this “country of genius in the data center”. That description you gave comes [00:48:57] from “Machines of Loving Grace”. I said we’ll get that in 2026, maybe 2027. Again, [00:49:02] that is my hunch. I wouldn’t be surprised if I’m off by a year or two, but that is my hunch. [00:49:08] Let’s say that happens. That’s the starting gun. How long does it take to cure all the diseases? [00:49:13] That’s one of the ways that drives a huge amount of economic value. You cure every disease. There’s [00:49:21] a question of how much of that goes to the pharmaceutical company or the AI company, [00:49:24] but there’s an enormous consumer surplus because —assuming we can get access for everyone, [00:49:29] which I care about greatly—we cure all of these diseases. How long does it take? You [00:49:34] have to do the biological discovery, you have to manufacture the new drug, [00:49:40] you have to go through the regulatory process. We saw this with vaccines and COVID. [00:49:47] We got the vaccine out to everyone, but it took a year and a half. [00:49:52] My question is: How long does it take to get the cure for everything—which AI is the genius [00:49:58] that can in theory invent—out to everyone? How long from when that AI first exists [00:50:03] in the lab to when diseases have actually been cured for everyone? [00:50:09] We’ve had a polio vaccine for 50 years. We’re still trying to eradicate it in the [00:50:14] most remote corners of Africa. The Gates Foundation is trying [00:50:18] as hard as they can. Others are trying as hard [00:50:20] as they can. But that’s difficult. Again, I don’t expect most of the economic diffusion [00:50:25] to be as difficult as that. That’s the most difficult case. But there’s a real dilemma here. [00:50:32] Where I’ve settled on it is that it will be faster than anything we’ve seen in the [00:50:39] world, but it still has its limits. So when we go to buying data centers, [00:50:47] again, the curve I’m looking at is: we’ve had a 10x a year increase every year. [00:50:54] At the beginning of this year, we’re looking at $10billioninannualizedrevenue.[00:51:02]Wehavetodecidehowmuchcomputetobuy.Ittakesayearortwotoactuallybuildout[00:51:10]thedatacenters,toreservethedatacenter.Basically{I}^{\prime }msaying,"In2027,[00:51:16]howmuchcomputedoIget?"Icouldassumethatthe[00:51:24]revenuewillcontinuegrowing10xayear,soi{t}^{\prime }llbe$100 billion at the end of [00:51:31] 2026 and $1trillionattheendof2027.Actuallyitwouldbe$5 trillion dollars [00:51:39] of compute because it would be $1trillionayearforfiveyears.[00:51:43]Icouldbuy$1 trillion of compute that starts at the end of 2027. [00:51:49] If my revenue is not $1trilliondollars,ifi{t}^{\prime }seven$800 billion, there’s no force on earth, [00:51:56] there’s no hedge on earth that could stop me from going bankrupt if I buy that much compute. [00:52:03] Even though a part of my brain wonders if it’s going to keep growing 10x, [00:52:07] I can’t buy $1trillionayearofcomputein2027.If{I}^{\prime }mjustoffbyayearinthatrateofgrowth,[00:52:17]orifthegrowthrateis5xayearinsteadof10xayear,thenyougobankrupt.[00:52:25]Soyouendupinaworldwhereyo{u}^{\prime }resupportinghundredsofbillions,nottrillions.[00:52:33]Youacceptsomeriskthatther{e}^{\prime }ssomuchdemandthatyouca{n}^{\prime }tsupporttherevenue,[00:52:38]andyouacceptsomeriskthatyougotitwrongandi{t}^{\prime }sstillslow.[00:52:43]WhenItalkedaboutbehavingresponsibly,whatImeantactuallywasnottheabsoluteamount.[00:52:51]Ithinkitistruew{e}^{\prime }respendingsomewhatlessthansomeoftheotherplayers.[00:52:55]I{t}^{\prime }sactuallytheotherthings,likehavewebeenthoughtfulaboutitorareweYOLOingandsaying,[00:53:01]"W{e}^{\prime }regoingtodo$100 billion here or $100billionthere"?[00:53:05]Igettheimpressionthatsomeoftheothercompanieshavenotwrittendown[00:53:09]thespreadsheet,thattheydo{n}^{\prime }treallyunderstandtherisksthe{y}^{\prime }retaking.[00:53:12]The{y}^{\prime }rejustdoingstuffbecauseitsoundscool.W{e}^{\prime }vethoughtcarefullyaboutit.W{e}^{\prime }re[00:53:19]anenterprisebusiness.Therefore,wecanrelymoreonrevenue.I{t}^{\prime }slessficklethanconsumer.[00:53:26]Wehavebettermargins,whichisthebufferbetweenbuyingtoomuchandbuyingtoolittle.[00:53:31]Ithinkweboughtanamountthatallowsustocaptureprettystrongupsideworlds.[00:53:37]Itwo{n}^{\prime }tcapturethefull10xayear.Thingswouldhavetogoprettybadlyfor[00:53:42]ustobeinfinancialtrouble.Sow{e}^{\prime }vethoughtcarefullyand[00:53:46]w{e}^{\prime }vemadethatbalance.Tha{t}^{\prime }swhatImeanwhen[00:53:48]Isaythatw{e}^{\prime }rebeingresponsible.Soitseemslikei{t}^{\prime }spossiblethatwe[00:53:54]actuallyjusthavedifferentdefinitionsofthe"countryofageniusinadatacenter".[00:53:56]BecausewhenIthinkofactualhumangeniuses,anactualcountryofhumangeniusesinadatacenter,[00:54:02]Iwouldhappilybuy$5 trillion worth of compute to run an actual country of [00:54:08] human geniuses in a data center. Let’s say JPMorgan or Moderna or [00:54:11] whatever doesn’t want to use them. I’ve got a country of geniuses. [00:54:14] They’ll start their own company. If they can’t start their own company and they’re bottlenecked [00:54:18] by clinical trials… It is worth stating that with clinical trials, most clinical trials fail because [00:54:22] the drug doesn’t work. There’s not efficacy. I make exactly that point in “Machines of [00:54:27] Loving Grace”, I say the clinical trials are going to go much faster [00:54:30] than we’re used to, but not infinitely fast. Okay, and then suppose it takes a year for [00:54:35] the clinical trials to work out so that you’re getting revenue from that and can make more drugs. [00:54:39] Okay, well, you’ve got a country of geniuses and you’re an AI lab. [00:54:44] You could use many more AI researchers. You also think there are these self-reinforcing [00:54:50] gains from smart people working on AI tech. You can have the data center [00:54:56] working on AI progress. Are there substantially [00:55:01] more gains from buying $1trillionayearofcomputeversus$300 billion a year of compute? [00:55:07] If your competitor is buying a trillion, yes there is. [00:55:09] Well, no, there’s some gain, but then again, there’s this chance that they go bankrupt before. [00:55:17] Again, if you’re off by only a year, you destroy yourselves. That’s the balance. We’re [00:55:23] buying a lot. We’re buying a hell of a lot. We’re buying an amount that’s comparable to [00:55:30] what the biggest players in the game are buying. But if you’re asking me, “Why haven’t we signed [00:55:39] $10trillionofcomputestartinginmid-2027?"...Firstofall,itca{n}^{\prime }tbeproduced.[00:55:44]Thereis{n}^{\prime }tthatmuchintheworld.Butsecond,whatifthecountryof[00:55:50]geniusescomes,butitcomesinmid-2028insteadofmid-2027?Yougobankrupt.[00:55:56]Soifyourprojectionisonetothreeyears,itseemslikeyoushouldwant[00:56:00]$10 trillion of compute by 2029 at the latest? Even in the longest version of the timelines [00:56:11] you state, the compute you are ramping up to build doesn’t seem in accordance. [00:56:16] What makes you think that? Human wages, let’s say, [00:56:21] are on the order of $50trillionayear—SoIwo{n}^{\prime }ttalkaboutAnthropicinparticular,[00:56:27]butifyoutalkabouttheindustry,theamountofcomputetheindustryisbuildingthisyearis[00:56:38]probably,callit,10-15gigawatts.Itgoesupbyroughly3xayear.[00:56:48]Sonextyea{r}^{\prime }s30-40gigawatts.2028mightbe100gigawatts.2029mightbelike300gigawatts.[00:57:03]I^{\prime }mdoingthemathinmyhead,buteachgigawattcostsmaybe$10 billion, [00:57:07] on the order of $10-15billionayear.Youputthatalltogetherandyo{u}^{\prime }re[00:57:14]gettingaboutwhatyoudescribed.You’regettingexactlythat.Yo{u}^{\prime }regettingmultiple[00:57:16]trillionsayearby2028or2029.Yo{u}^{\prime }regettingexactlywhatyoupredict.[00:57:23]Tha{t}^{\prime }sfortheindustry.Tha{t}^{\prime }sfortheindustry,that’sright.[00:57:26]SupposeAnthropi{c}^{\prime }scomputekeeps3x-ingayear,andthenby2027-28,youhave10gigawatts.[00:57:34]Multiplythatby,asyousay,$10 billion. So then it’s like $100billionayear.[00:57:40]Butthenyo{u}^{\prime }resayingtheTAMby2028is$200 billion. [00:57:43] Again, I don’t want to give exact numbers for Anthropic, but these numbers are too small. [00:57:48] Okay, interesting. You’ve told investors [00:58:49] that you plan to be profitable starting in 2028. This is the year when we’re potentially getting [00:58:55] the country of geniuses as a data center. This is now going to unlock all this progress [00:59:02] in medicine and health and new technologies. Wouldn’t this be exactly the time where you’d [00:59:11] want to reinvest in the business and build bigger “countries” so they can make more discoveries? [00:59:16] Profitability is this kind of weird thing in this field. [00:59:21] I don’t think in this field profitability is actually a measure of spending down [00:59:32] versus investing in the business. Let’s just take a model of this. [00:59:36] I actually think profitability happens when you underestimated the amount of demand you were going [00:59:41] to get and loss happens when you overestimated the amount of demand you were going to get, [00:59:46] because you’re buying the data centers ahead of time. Think about it this way. Again, [00:59:52] these are stylized facts. These numbers are not exact. I’m just trying to make a toy model here. [00:59:56] Let’s say half of your compute is for training and half of your compute is for inference. [01:00:02] The inference has some gross margin that’s more than 50%. [01:00:07] So what that means is that if you were in steady-state, you build a data center and if [01:00:12] you knew exactly the demand you were getting, you would get a certain amount of revenue. [01:00:23] Let’s say you pay $100billionayearforcompute.On$50 billion a year you support [01:00:28] $150billionofrevenue.Theother$50 billion is used for training. [01:00:36] Basically you’re profitable and you make $50billionofprofit.[01:00:40]Thosearetheeconomicsoftheindustrytoday,ornottodaybutwherewe’re[01:00:45]projectingforwardinayearortwo.Theonlythingthatmakesthatnotthe[01:00:49]caseisifyougetlessdemandthan$50 billion. Then you have more than 50% of your data center [01:00:57] for research and you’re not profitable. So you train stronger models, [01:01:01] but you’re not profitable. If you get more demand than you thought, then [01:01:07] research gets squeezed, but you’re kind of able to support more inference and you’re more profitable. [01:01:16] Maybe I’m not explaining it well, but the thing I’m trying to say is that you [01:01:19] decide the amount of compute first. Then you have some target desire of [01:01:24] inference versus training, but that gets determined by demand. [01:01:28] It doesn’t get determined by you. What I’m hearing is the reason [01:01:30] you’re predicting profit is that you are systematically underinvesting in compute? [01:01:37] No, no, no. I’m saying it’s hard to predict. These things about 2028 and when it will happen, [01:01:43] that’s our attempt to do the best we can with investors. [01:01:46] All of this stuff is really uncertain because of the cone of uncertainty. [01:01:50] We could be profitable in 2026 if the revenue grows fast enough. [01:01:58] If we overestimate or underestimate the next year, that could swing wildly. [01:02:04] What I’m trying to get at is that you have a model in your head of a business that invests, [01:02:09] invests, invests, gets scale and then becomes profitable. [01:02:14] There’s a single point at which things turn around. [01:02:16] I don’t think the economics of this industry work that way. [01:02:19] I see. So if I’m understanding correctly, you’re saying that because of the discrepancy [01:02:24] between the amount of compute we should have gotten and the amount of compute we got, [01:02:27] we were sort of forced to make profit. But that doesn’t mean we’re going [01:02:30] to continue making profit. We’re going to reinvest the money [01:02:33] because now AI has made so much progress and we want a bigger country of geniuses. [01:02:37] So back into revenue is high, but losses are also high. [01:02:44] If every year we predict exactly what the demand is going to be, we’ll be profitable every year. [01:02:50] Because spending 50% of your compute on research, roughly, plus a gross margin that’s higher than [01:03:00] 50% and correct demand prediction leads to profit. That’s the profitable business model that I think [01:03:07] is kind of there, but obscured by these building ahead and prediction errors. [01:03:13] I guess you’re treating the 50% as a sort of given constant, whereas in fact, [01:03:21] if AI progress is fast and you can increase the progress by scaling up more, you should just have [01:03:24] more than 50% and not make profit. But here’s what I’ll say. You [01:03:26] might want to scale it up more. Remember the log returns to scale. [01:03:34] If 70% would get you a very little bit of a smaller model through a factor of 1.4x… [01:03:42] That extra $20billion,eachdollarthereisworthmuchlesstoyoubecauseofthelog-linearsetup.[01:03:51]Soyoumightfindthati{t}^{\prime }sbettertoinvestthat$20 billion in serving [01:03:58] inference or in hiring engineers who are kind of better at what they’re doing. [01:04:05] So the reason I said 50%… That’s not exactly our target. It’s not exactly going to be 50%. [01:04:10] It’ll probably vary over time. What I’m saying is the log-linear return, what it leads to is you [01:04:18] spend of order one fraction of the business. Like not 5%, not 95%. Then you get diminishing returns. [01:04:28] I feel strange that I’m convincing Dario to believe in AI progress or something. [01:04:34] Okay, you don’t invest in research because it has diminishing returns, [01:04:37] but you invest in the other things you mentioned. I think profit at a sort of macro level— [01:04:39] Again, I’m talking about diminishing returns, but after you’re spending 50 billion a year. [01:04:46] This is a point I'm sure you would make, but diminishing returns on a genius could [01:04:51] be quite high. More generally, [01:04:54] what is profit in a market economy? Profit is basically saying other [01:04:58] companies in the market can do more things with this money than I can. [01:05:02] Put aside Anthropic. I don't want to give information about Anthropic. [01:05:06] That’s why I'm giving these stylized numbers. But let's just derive the [01:05:10] equilibrium of the industry. Why doesn't everyone spend 100% of their [01:05:21] compute on training and not serve any customers? It's because if they didn't get any revenue, [01:05:25] they couldn't raise money, they couldn't do compute deals, [01:05:27] they couldn't buy more compute the next year. So there's going to be an equilibrium where every [01:05:31] company spends less than 100% on training and certainly less than 100% on inference. [01:05:38] It should be clear why you don't just serve the current models and never train another model, [01:05:44] because then you don't have any demand because you'll fall behind. So there's some equilibrium. [01:05:49] It's not gonna be 10%, it's not gonna be 90%. Let's just say as a stylized fact, it's 50%. [01:05:55] That's what I'm getting at. I think we're gonna be in a position where that equilibrium of how much [01:06:01] you spend on training is less than the gross margins that you're able to get on compute. [01:06:08] So the underlying economics are profitable. The problem is you have this hellish demand [01:06:14] prediction problem when you're buying the next year of compute and you might guess under and be [01:06:21] very profitable but have no compute for research. Or you might guess over and you are not [01:06:30] profitable and you have all the compute for research in the world. Does that make sense? [01:06:36] Just as a dynamic model of the industry? Maybe stepping back, I'm not saying I think [01:06:42] the "country of geniuses" is going to come in two years and therefore you should buy this compute. [01:06:47] To me, the end conclusion you're arriving at makes a lot of sense. [01:06:51] But that's because it seems like "country of geniuses" is hard and there's a long way to go. [01:06:57] So stepping back, the thing I'm trying to get at is more that it seems like your worldview [01:07:03] is compatible with somebody who says, "We're like 10 years away from a world in which we're [01:07:07] generating trillions of dollars of value." That's just not my view. So I'll make [01:07:14] another prediction. It is hard for me to see that there won't be trillions [01:07:20] of dollars in revenue before 2030. I can construct a plausible world. [01:07:26] It takes maybe three years. That would be the end of what I think it's plausible. [01:07:31] Like in 2028, we get the real "country of geniuses in the data center". [01:07:36] The revenue's going into the low hundreds of billions by 2028, and then the country [01:07:46] of geniuses accelerates it to trillions. We’re basically on the slow end of diffusion. [01:07:52] It takes two years to get to the trillions. That would be the world where it takes until 2030. [01:07:59] I suspect even composing the technical exponential and diffusion exponential, [01:08:05] we’ll get there before 2030. So you laid out a model where Anthropic makes [01:08:10] profit because it seems like fundamentally we're in a compute-constrained world. [01:08:14] So eventually we keep growing compute— I think the way the profit comes is… Again, [01:08:21] let's just abstract the whole industry here. Let's just imagine we're in an economics textbook. [01:08:27] We have a small number of firms. Each can invest a limited amount. [01:08:33] Each can invest some fraction in R&D. They have some marginal cost to serve. [01:08:38] The gross profit margins on that marginal cost are very high because inference is efficient. [01:08:47] There's some competition, but the models are also differentiated. [01:08:52] Companies will compete to push their research budgets up. [01:08:55] But because there's a small number of players, we have the... What is it called? [01:09:00] The Cournot equilibrium, I think, is what the small number of firm equilibrium is. [01:09:05] The point is it doesn't equilibrate to perfect competition with zero margins. [01:09:15] If there's three firms in the economy and all are kind of independently behaving rationally, [01:09:20] it doesn't equilibrate to zero. Help me understand that, because [01:09:24] right now we do have three leading firms and they're not making profit. So what is changing? [01:09:33] Again, the gross margins right now are very positive. [01:09:38] What's happening is a combination of two things. One is that we're still in the exponential [01:09:43] scale-up phase of compute. A model gets trained. Let's say a model got [01:09:53] trained that costs 1 billion last year. Then this year it produced $4billionof[01:10:02]revenueandcost$1 billion to inference from. Again, I’m using stylized numbers here, but that [01:10:12] would be 75% gross margins and this 25% tax. So that model as a whole makes $2billion.[01:10:23]Butatthesametime,w{e}^{\prime }respending$10 billion to train the next model because [01:10:27] there’s an exponential scale-up. So the company loses money. Each model [01:10:31] makes money, but the company loses money. The equilibrium I’m talking about is an [01:10:35] equilibrium where we have the “country of geniuses in a data center”, but that [01:10:43] model training scale-up has equilibrated more. Maybe it’s still going up. We’re still trying to [01:10:49] predict the demand, but it’s more leveled out. I’m confused about a couple of things there. [01:10:56] Let’s start with the current world. In the current world, you’re right that, [01:11:00] as you said before, if you treat each individual model as a company, it’s profitable. [01:11:05] But of course, a big part of the production function of being a frontier lab is training [01:11:11] the next model, right? Yes, that’s right. [01:11:13] If you didn’t do that, then you’d make profit for two months and then [01:11:16] you wouldn’t have margins because you wouldn’t have the best model. [01:11:19] But at some point that reaches the biggest scale that it can reach. [01:11:23] And then in equilibrium, we have algorithmic improvements, but we’re spending roughly the [01:11:28] same amount to train the next model as we spend to train the current model. [01:11:37] At some point you run out of money in the economy. A fixed lump of labor fallacy… The economy is [01:11:42] going to grow, right? That’s one of your predictions. We’re going [01:11:44] to have the data centers in space. Yes, but this is another example [01:11:47] of the theme I was talking about. The economy will grow much faster [01:11:53] with AI than I think it ever has before. Right now the compute is growing 3x a year. [01:11:59] I don’t believe the economy is gonna grow 300% a year. [01:12:03] I said this in “Machines of Loving Grace”, I think we may get 10-20% [01:12:08] per year growth in the economy, but we’re not gonna get 300% growth in the economy. [01:12:13] So I think in the end, if compute becomes the majority of what the economy produces, [01:12:18] it’s gonna be capped by that. So let’s assume a model [01:12:22] where compute stays capped. The world where frontier labs are making money [01:12:26] is one where they continue to make fast progress. Because fundamentally your margin is limited by [01:12:34] how good the alternative is. So you are able to make money [01:12:37] because you have a frontier model. If you didn’t have a frontier model [01:12:39] you wouldn’t be making money. So this model requires there [01:12:45] never to be a steady state. Forever and ever you keep [01:12:48] making more algorithmic progress. I don’t think that’s true. I mean, [01:12:51] I feel like we’re in an economics class. Do you know the Tyler Cowen quote? [01:12:59] We never stop talking about economics. We never stop talking about economics. [01:13:03] So no, I don’t think this field’s going to be a monopoly. [01:13:12] All my lawyers never want me to say the word “monopoly”. [01:13:15] But I don’t think this field’s going to be a monopoly. [01:13:17] You do get industries in which there are a small number of players. [01:13:21] Not one, but a small number of players. Ordinarily, the way you get monopolies [01:13:27] like Facebook or Meta—I always call them Facebook—is these kinds of network effects. [01:13:37] The way you get industries in which there are a small number of players, [01:13:41] is very high costs of entry. Cloud is like this. I think cloud is a good example of this. [01:13:49] There are three, maybe four, players within cloud. I think that’s the same for AI, three, maybe four. [01:13:56] The reason is that it’s so expensive. It requires so much expertise and so [01:14:02] much capital to run a cloud company. You have to put up all this capital. [01:14:08] In addition to putting up all this capital, you have to get all of this other stuff [01:14:11] that requires a lot of skill to make it happen. So if you go to someone and you’re like, “I want [01:14:17] to disrupt this industry, here’s $100billion."Yo{u}^{\prime }relike,"okay,I^{\prime }mputtingin$100 billion [01:14:22] and also betting that you can do all these other things that these people have been doing.” [01:14:26] Only to decrease the profit. The effect of your entering [01:14:29] is that profit margins go down. So, we have equilibria like this [01:14:33] all the time in the economy where we have a few players. Profits are not astronomical. Margins [01:14:39] are not astronomical, but they’re not zero. That’s what we see on cloud. Cloud is very [01:14:47] undifferentiated. Models are more differentiated than cloud. [01:14:51] Everyone knows Claude is good at different things than GPT is good at, than Gemini is good at. [01:14:58] It’s not just that Claude’s good at coding, GPT is good at math and reasoning. [01:15:05] It’s more subtle than that. Models are good at different types of coding. Models have different [01:15:09] styles. I think these things are actually quite different from each other, and so I would expect [01:15:15] more differentiation than you see in cloud. Now, there actually is one counter-argument. [01:15:26] That counter-argument is if the process of producing models, [01:15:32] if AI models can do that themselves, then that could spread throughout the economy. [01:15:37] But that is not an argument for commoditizing AI models in general. [01:15:41] That’s kind of an argument for commoditizing the whole economy at once. [01:15:45] I don’t know what quite happens in that world where basically anyone [01:15:48] can do anything, anyone can build anything, and there’s no moat around anything at all. [01:15:53] I don’t know, maybe we want that world. Maybe that’s the end state here. [01:15:58] Maybe when AI models can do everything, if we’ve solved all the safety and security problems, [01:16:09] that’s one of the mechanisms for the economy just flattening itself again. [01:16:17] But that’s kind of far post-”country of geniuses in the data center.” [01:16:23] Maybe a finer way to put that potential point is: 1) it seems like AI research is especially [01:16:32] loaded on raw intellectual power, which will be especially abundant in the world of AGI. [01:16:37] And 2) if you just look at the world today, there are very few technologies that seem to be [01:16:41] diffusing as fast as AI algorithmic progress. So that does hint that this industry is [01:16:50] sort of structurally diffusive. I think coding is going fast, but [01:16:54] I think AI research is a superset of coding and there are aspects of it that are not going fast. [01:17:00] But I do think, again, once we get coding, once we get AI models going fast, then that will speed up [01:17:07] the ability of AI models to do everything else. So while coding is going fast now, I think once [01:17:13] the AI models are building the next AI models and building everything else, [01:17:17] the whole economy will kind of go at the same pace. I am worried geographically, though. [01:17:24] I’m a little worried that just proximity to AI, having heard about AI, may be one differentiator. [01:17:34] So when I said the 10-20% growth rate, a worry I have is that the growth rate could be like 50% [01:17:42] in Silicon Valley and parts of the world that are socially connected to Silicon Valley, and not that [01:17:50] much faster than its current pace elsewhere. I think that’d be a pretty messed up world. [01:17:54] So one of the things I think about a lot is how to prevent that. [01:17:57] Do you think that once we have this country of geniuses in a data center, that [01:18:01] robotics is sort of quickly solved afterwards? Because it seems like a big problem with robotics [01:18:06] is that a human can learn how to teleoperate current hardware, but current AI models can’t, [01:18:12] at least not in a way that’s super productive. And so if we have this ability to learn like [01:18:16] a human, shouldn’t it solve robotics immediately as well? [01:18:19] I don’t think it’s dependent on learning like a human. [01:18:21] It could happen in different ways. Again, we could have trained the model on [01:18:25] many different video games, which are like robotic controls, or many different simulated robotics [01:18:30] environments, or just train them to control computer screens, and they learn to generalize. [01:18:34] So it will happen… it’s not necessarily dependent on human-like learning. [01:18:41] Human-like learning is one way it could happen. If the model’s like, “Oh, I pick up a robot, [01:18:44] I don’t know how to use it, I learn,” that could happen because we discovered continual learning. [01:18:50] That could also happen because we trained the model on a bunch of environments and [01:18:54] then generalized, or it could happen because the model learns that in the context length. [01:18:58] It doesn’t actually matter which way. If we go back to the discussion we had [01:19:03] an hour ago, that type of thing can happen in several different ways. [01:19:10] But I do think when for whatever reason the models have those skills, then robotics will be [01:19:16] revolutionized—both the design of robots, because the models will be much better than humans at [01:19:21] that, and also the ability to control robots. So we’ll get better at building the physical [01:19:28] hardware, building the physical robots, and we’ll also get better at controlling it. [01:19:32] Now, does that mean the robotics industry will also be generating [01:19:36] trillions of dollars of revenue? My answer there is yes, but there will be [01:19:40] the same extremely fast, but not infinitely fast diffusion. So will robotics be revolutionized? [01:19:46] Yeah, maybe tack on another year or two. That’s the way I think about these things. [01:19:52] Makes sense. There’s a general skepticism about extremely fast progress. Here’s my view. It sounds [01:19:58] like you are going to solve continual learning one way or another within a matter of years. [01:20:02] But just as people weren’t talking about continual learning a couple of years ago, [01:20:06] and then we realized, “Oh, why aren’t these models as useful as they could be right now, [01:20:09] even though they are clearly passing the Turing test and are experts in so many different domains? [01:20:14] Maybe it’s this thing.” Then we solve this thing and we realize, actually, there’s another thing [01:20:19] that human intelligence can do that’s a basis of human labor that these models can’t do. [01:20:24] So why not think there will be more things like this, where [01:20:28] we’ve found more pieces of human intelligence? Well, to be clear, I think continual learning, as [01:20:33] I’ve said before, might not be a barrier at all. I think we may just get there by pre-training [01:20:40] generalization and RL generalization. I think there just [01:20:48] might not be such a thing at all. In fact, I would point to the history [01:20:51] in ML of people coming up with things that are barriers that end up kind of [01:20:56] dissolving within the big blob of compute. People talked about, “How do your models [01:21:06] keep track of nouns and verbs?” “They can understand syntactically, [01:21:11] but they can’t understand semantically? It’s only statistical correlations.” [01:21:16] “You can understand a paragraph, you can’t understand a word. [01:21:19] There’s reasoning, you can’t do reasoning.” But then suddenly it turns out you can [01:21:23] do code and math very well. So I think there’s actually a [01:21:27] stronger history of some of these things seeming like a big deal and then kind of dissolving. Some [01:21:35] of them are real. The need for data is real, maybe continual learning is a real thing. [01:21:42] But again, I would ground us in something like code. [01:21:46] I think we may get to the point in a year or two where the models can [01:21:50] just do SWE end-to-end. That’s a whole task. That’s a whole sphere of human activity that [01:21:56] we’re just saying models can do now. When you say end-to-end, do you mean [01:22:02] setting technical direction, understanding the context of the problem, et cetera? [01:22:06] Yes. I mean all of that. Interesting. I feel like that is AGI-complete, [01:22:13] which maybe is internally consistent. But it’s not like saying 90% [01:22:17] of code or 100% of code. No, I gave this spectrum: [01:22:22] 90% of code, 100% of code, 90% of end-to-end SWE, 100% of end-to-end SWE. [01:22:28] New tasks are created for SWEs. Eventually those get done as well. [01:22:31] It’s a long spectrum there, but we’re traversing the spectrum very quickly. [01:22:35] I do think it’s funny that I’ve seen a couple of podcasts you’ve done where [01:22:40] the hosts will be like, “But Dwarkesh wrote the essay about the continuous learning thing.” [01:22:43] It always makes me crack up because you’ve been an AI researcher for 10 years. [01:22:48] I’m sure there’s some feeling of, “Okay, so a podcaster wrote an essay, [01:22:53] and every interview I get asked about it.” The truth of the matter is that we’re all [01:22:59] trying to figure this out together. There are some ways in which I’m [01:23:04] able to see things that others aren’t. These days that probably has more to do [01:23:08] with seeing a bunch of stuff within Anthropic and having to make a bunch of decisions than I have [01:23:13] any great research insight that others don’t. I’m running a 2,500 person company. [01:23:20] It’s actually pretty hard for me to have concrete research insight, much harder than it would have [01:23:27] been 10 years ago or even two or three years ago. As we go towards a world of a full drop-in [01:23:36] remote worker replacement, does an API pricing model still make the most sense? [01:23:42] If not, what is the correct way to price AGI, or serve AGI? [01:23:45] I think there’s going to be a bunch of different business models here, all at once, [01:23:49] that are going to be experimented with. I actually do think that the API [01:23:59] model is more durable than many people think. One way I think about it is if the technology [01:24:06] is advancing quickly, if it’s advancing exponentially, what that means is there’s [01:24:12] always a surface area of new use cases that have been developed in the last three months. [01:24:20] Any kind of product surface you put in place is always at risk of sort of becoming irrelevant. [01:24:27] Any given product surface probably makes sense for a range of capabilities of the model. [01:24:32] The chatbot is already running into limitations where making it smarter doesn’t really help the [01:24:39] average consumer that much. But I don’t think that’s [01:24:41] a limitation of AI models. I don’t think that’s evidence [01:24:45] that the models are good enough and them getting better doesn’t matter to the economy. [01:24:51] It doesn’t matter to that particular product. So I think the value of the API is that the API [01:24:58] always offers an opportunity, very close to the bare metal, to build on what the latest thing is. [01:25:06] There’s always going to be this front of new startups and new ideas that [01:25:14] weren’t possible a few months ago and are possible because the model is advancing. [01:25:19] I actually predict that it’s going to exist alongside other models, but we’re always going [01:25:28] to have the API business model because there’s always going to be a need for a thousand different [01:25:34] people to try experimenting with the model in a different way. 100 of them become startups and [01:25:40] ten of them become big successful startups. Two or three really end up being the way [01:25:45] that people use the model of a given generation. So I basically think it’s always going to exist. [01:25:50] At the same time, I’m sure there’s going to be other models as well. [01:25:55] Not every token that’s output by the model is worth the same amount. [01:26:00] Think about what is the value of the tokens that the model outputs when someone calls [01:26:10] them up and says, “My Mac isn’t working,” or something, the model’s like, “restart it.” [01:26:16] Someone hasn’t heard that before, but the model said that 10 million times. [01:26:23] Maybe that’s worth like a dollar or a few cents or something. [01:26:26] Whereas if the model goes to one of the pharmaceutical companies and it says, “Oh, [01:26:34] you know, this molecule you’re developing, you should take the aromatic ring from that end of the [01:26:39] molecule and put it on that end of the molecule. If you do that, wonderful things will happen.” [01:26:46] Those tokens could be worth tens of millions of dollars. [01:26:52] So I think we’re definitely going to see business models that recognize that. [01:26:56] At some point we’re going to see “pay for results” in some form, or we may see forms of compensation [01:27:06] that are like labor, that kind of work by the hour. I don’t know. I think because it’s a new [01:27:16] industry, a lot of things are going to be tried. I don’t know what will turn out to [01:27:19] be the right thing. I take your point that [01:27:24] people will have to try things to figure out what is the best way to use this blob of intelligence. [01:27:28] But what I find striking is Claude Code. I don’t think in the history of startups [01:27:34] there has been a single application that has been as hotly competed in as coding agents. [01:27:42] Claude Code is a category leader here. That seems surprising to me. It doesn’t seem [01:27:49] intrinsically that Anthropic had to build this. I wonder if you have an accounting of why it had [01:27:54] to be Anthropic or how Anthropic ended up building an application in addition [01:27:58] to the model underlying it that was successful. So it actually happened in a pretty simple way, [01:28:02] which is that we had our own coding models, which were good at coding. [01:28:09] Around the beginning of 2025, I said, “I think the time has come where you can have [01:28:14] nontrivial acceleration of your own research if you’re an AI company by using these models.” [01:28:21] Of course, you need an interface, you need a harness to use them. [01:28:25] So I encouraged people internally. I didn’t say this is one thing that you have to use. [01:28:31] I just said people should experiment with this. I think it might have been originally [01:28:37] called Claude CLI, and then the name eventually got changed to Claude Code. [01:28:42] Internally, it was the thing that everyone was using and it was seeing fast internal adoption. [01:28:48] I looked at it and I said, “Probably we should launch this externally, right?” [01:28:53] It’s seen such fast adoption within Anthropic. Coding is a lot of what we do. [01:28:59] We have an audience of many, many hundreds of people that’s in some ways at least [01:29:04] representative of the external audience. So it looks like we already have product [01:29:08] market fit. Let’s launch this thing. And then we launched it. I think just the fact that we [01:29:15] ourselves are kind of developing the model and we ourselves know what we most need to use the model, [01:29:21] I think it’s kind of creating this feedback loop. I see. In the sense that you, let’s say a [01:29:26] developer at Anthropic is like, “Ah, it would be better if it was better at this X thing.” [01:29:31] Then you bake that into the next model that you build. [01:29:35] That’s one version of it, but then there’s just the ordinary product iteration. [01:29:41] We have a bunch of coders within Anthropic, they use Claude Code [01:29:47] every day and so we get fast feedback. That was more important in the early days. [01:29:50] Now, of course, there are millions of people using it, and so we get [01:29:53] a bunch of external feedback as well. But it’s just great to be able to get [01:29:58] kind of fast internal feedback. I think this is the reason why we [01:30:03] launched a coding model and didn’t launch a pharmaceutical company. [01:30:10] My background’s in biology, but we don’t have any of the resources that [01:30:14] are needed to launch a pharmaceutical company. Let me now ask you about making AI go well. [01:31:24] It seems like whatever vision we have about how AI goes well has to be compatible with two things: [01:31:30] 1) the ability to build and run AIs is diffusing extremely rapidly and 2) the [01:31:37] population of AIs, the amount we have and their intelligence, will also increase very rapidly. [01:31:44] That means that lots of people will be able to build huge populations of misaligned AIs, [01:31:49] or AIs which are just companies which are trying to increase their [01:31:53] footprint or have weird psyches like Sydney Bing, but now they’re superhuman. [01:31:57] What is a vision for a world in which we have an equilibrium that is compatible [01:32:02] with lots of different AIs, some of which are misaligned, running around? [01:32:06] I think in “The Adolescence of Technology”, I was skeptical of the balance of power. [01:32:13] But the thing I was specifically skeptical of is you have three or four of these companies [01:32:23] all building models that are derived from the same thing, that they would check each other. [01:32:36] Or even that any number of them would check each other. [01:32:40] We might live in an offense-dominant world where one person or one AI model is smart enough to do [01:32:47] something that causes damage for everything else. In the short run, we have a limited number [01:32:54] of players now. So we can start [01:32:56] within the limited number of players. We need to put in place the safeguards. [01:33:03] We need to make sure everyone does the right alignment work. [01:33:05] We need to make sure everyone has bioclassifiers. Those are the immediate things we need to do. [01:33:11] I agree that that doesn’t solve the problem in the long run, particularly if the ability of [01:33:16] AI models to make other AI models proliferates, then the whole thing can become harder to solve. [01:33:26] I think in the long run we need some architecture of governance. [01:33:30] We need some architecture of governance that preserves human freedom, [01:33:35] but also allows us to govern a very large number of human systems, AI systems, hybrid [01:33:52] human-AI companies or economic units. So we’re gonna need to think about: [01:34:01] how do we protect the world against bioterrorism? How do we protect the world against mirror life? [01:34:11] Probably we’re gonna need some kind of AI monitoring system [01:34:15] that monitors for all of these things. But then we need to build this in a way [01:34:20] that preserves civil liberties and our constitutional rights. [01:34:24] So I think just as anything else, it’s a new security landscape with a new set of [01:34:34] tools and a new set of vulnerabilities. My worry is, if we had 100 years for this [01:34:40] to happen all very slowly, we’d get used to it. We’ve gotten used to the presence of explosives [01:34:49] in society or the presence of various new weapons or the presence of video cameras. [01:34:58] We would get used to it over 100 years and we’d develop governance mechanisms. We’d [01:35:03] make our mistakes. My worry is just that this is happening all so fast. [01:35:07] So maybe we need to do our thinking faster about how to make these governance mechanisms work. [01:35:13] It seems like in an offense-dominant world, over the course of the next century—the idea is that AI [01:35:19] is making the progress that would happen over the next century happen in some period of five to ten [01:35:22] years—we would still need the same mechanisms, or balance of power would be similarly intractable, [01:35:29] even if humans were the only game in town. I guess we have the advice of AI. [01:35:36] But it fundamentally doesn’t seem like a totally different ball game here. [01:35:41] If checks and balances were going to work, they would work with humans as well. [01:35:44] If they aren’t going to work, they wouldn’t work with AIs as well. [01:35:47] So maybe this just dooms human checks and balances as well. [01:35:51] Again, I think there’s some way to make this happen. [01:35:58] The governments of the world may have to work together to make it happen. [01:36:02] We may have to talk to AIs about building societal structures in such a way that these [01:36:10] defenses are possible. I don’t know. I don’t want to say this is so far ahead in time, [01:36:15] but it’s so far ahead in technological ability that may happen over a short period of time, [01:36:21] that it’s hard for us to anticipate it in advance. Speaking of governments getting involved, [01:36:25] on December 26, the Tennessee legislature introduced a bill which said, “It would [01:36:31] be an offense for a person to knowingly train artificial intelligence to provide [01:36:34] emotional support, including through open-ended conversations with a user.” [01:36:39] Of course, one of the things that Claude attempts to do is be a thoughtful, knowledgeable friend. [01:36:48] In general, it seems like we’re going to have this patchwork of state laws. [01:36:52] A lot of the benefits that normal people could experience as a result of AI are going to be [01:36:56] curtailed, especially when we get into the kinds of things you discuss in “Machines [01:36:59] of Loving Grace”: biological freedom, mental health improvements, et cetera. [01:37:02] It seems easy to imagine worlds in which these get Whac-A-Moled away by different laws, whereas [01:37:10] bills like this don’t seem to address the actual existential threats that you’re concerned about. [01:37:15] I’m curious to understand, in the context of things like this, Anthropic’s position [01:37:20] against the federal moratorium on state AI laws. There are many different things going on at once. [01:37:28] I think that particular law is dumb. It was clearly made by legislators [01:37:34] who just probably had little idea what AI models could do and not do. [01:37:38] They’re like, “AI models serving us, that just sounds scary. [01:37:41] I don’t want that to happen.” So we’re not in favor of that. [01:37:47] But that wasn’t the thing that was being voted on. The thing that was being voted on is: [01:37:52] we’re going to ban all state regulation of AI for 10 years with no apparent plan to do any [01:38:00] federal regulation of AI, which would take Congress to pass, which is a very high bar. [01:38:05] So the idea that we’d ban states from doing anything for 10 years… People said they had [01:38:11] a plan for the federal government, but there was no actual proposal on the table. There was [01:38:15] no actual attempt. Given the serious dangers that I lay out in “Adolescence of Technology” [01:38:22] around things like biological weapons and bioterrorism autonomy risk, and the [01:38:29] timelines we’ve been talking about—10 years is an eternity—I think that’s a crazy thing to do. [01:38:36] So if that’s the choice, if that’s what you force us to choose, then we’re going [01:38:42] to choose not to have that moratorium. I think the benefits of that position [01:38:47] exceed the costs, but it’s not a perfect position if that’s the choice. [01:38:51] Now, I think the thing that we should do, the thing that I would support, is the federal [01:38:56] government should step in, not saying “states you can’t regulate”, but “Here’s what we’re going to [01:39:02] do, and states you can’t differ from this.” I think preemption is fine in the sense of [01:39:08] saying that the federal government says, “Here is our standard. This applies to everyone. [01:39:12] States can’t do something different.” That would be something I would support [01:39:16] if it would be done in the right way. But this idea of states, “You can’t do [01:39:22] anything and we’re not doing anything either,” that struck us as very much not making sense. [01:39:29] I think it will not age well, it is already starting to not age well with [01:39:33] all the backlash that you’ve seen. Now, in terms of what we would want, [01:39:39] the things we’ve talked about are starting with transparency standards in order to monitor some [01:39:46] of these autonomy risks and bioterrorism risks. As the risks become more serious, as we get more [01:39:53] evidence for them, then I think we could be more aggressive in some targeted ways and say, “Hey, [01:39:58] AI bioterrorism is really a threat. Let’s pass a law that forces [01:40:04] people to have classifiers.” I could even imagine… It depends. [01:40:07] It depends how serious the threat it ends up being. We don’t know for sure. We need to pursue [01:40:12] this in an intellectually honest way where we say that ahead of time, the risk has not emerged yet. [01:40:16] But I could certainly imagine, with the pace that things are going at, [01:40:21] a world where later this year we say, “Hey, this AI bioterrorism stuff is really serious. [01:40:27] We should do something about it. We should put it in a federal standard. [01:40:31] If the federal government won’t act, we should put it in a state standard.” I could totally see that. [01:40:36] I’m concerned about a world where if you just consider the pace of progress you’re expecting, [01:40:42] the life cycle of legislation… The benefits are, as you say because [01:40:48] of diffusion lag, slow enough that I really do think this patchwork of state [01:40:55] laws, on the current trajectory, would prohibit. I mean if having an emotional chatbot friend is [01:40:59] something that freaks people out, then just imagine the kinds of actual benefits from AI [01:41:03] we want normal people to be able to experience. From improvements in health and healthspan and [01:41:08] improvements in mental health and so forth. Whereas at the same time, it seems like you [01:41:13] think the dangers are already on the horizon and I just don’t see that much… It seems like it would [01:41:19] be especially injurious to the benefits of AI as compared to the dangers of AI. [01:41:24] So that’s maybe where the cost benefit makes less sense to me. [01:41:27] So there’s a few things here. People talk about there being [01:41:31] thousands of these state laws. First of all, the vast, [01:41:34] vast majority of them do not pass. The world works a certain way in theory, [01:41:41] but just because a law has been passed doesn’t mean it’s really enforced. [01:41:44] The people implementing it may be like, “Oh my God, this is stupid. [01:41:48] It would mean shutting off everything that’s ever been built in Tennessee.” [01:41:55] Very often, laws are interpreted in a way that makes them not as dangerous or harmful. [01:42:02] On the same side, of course, you have to worry if you’re passing a law to stop a bad thing; [01:42:06] you have this problem as well. My basic view is that if we could [01:42:16] decide what laws were passed and how things were done—and we’re only one small input [01:42:21] into that—I would deregulate a lot of the stuff around the health benefits of AI. [01:42:29] I don’t worry as much about the chatbot laws. I actually worry more about the drug approval [01:42:37] process, where I think AI models are going to greatly accelerate the rate at which we discover [01:42:45] drugs, and the pipeline will get jammed up. The pipeline will not be prepared to process [01:42:50] all the stuff that’s going through it. I think reform of the regulatory process [01:42:58] should bias more towards the fact that we have a lot of things coming where the safety and [01:43:02] efficacy is actually going to be really crisp and clear, a beautiful thing, and really effective. [01:43:12] Maybe we don’t need all this superstructure around it that was designed around an era of drugs that [01:43:21] barely work and often have serious side effects. At the same time, I think we should be [01:43:26] ramping up quite significantly the safety and security legislation. [01:43:35] Like I’ve said, starting with transparency is my view of trying not to hamper the industry, [01:43:43] trying to find the right balance. I’m worried about it. Some people criticize [01:43:46] my essay for saying, “That’s too slow. The dangers of AI will come too soon [01:43:50] if we do that.” Well, basically, [01:43:52] I think the last six months and maybe the next few months are going to be about transparency. [01:43:58] Then, if these risks emerge when we’re more certain of them—which [01:44:02] I think we might be as soon as later this year—then I think we need to act very fast [01:44:07] in the areas where we’ve actually seen the risk. I think the only way to do this is to be nimble. [01:44:13] Now, the legislative process is normally not nimble, but we need to emphasize the [01:44:21] urgency of this to everyone involved. That’s why I’m sending this message of urgency. [01:44:24] That’s why I wrote Adolescence of Technology. I wanted policymakers, economists, national [01:44:30] security professionals, and decision-makers to read it so that they have some hope of acting [01:44:36] faster than they would have otherwise. Is there anything you can do or advocate [01:44:42] that would make it more certain that the benefits of AI are better instantiated? [01:44:51] I feel like you have worked with legislatures to say, “Okay, [01:44:54] we’re going to prevent bioterrorism here. We’re going to increase transparency, we’re [01:44:57] going to increase whistleblower protection.” But I think by default, the actual benefits [01:45:01] we’re looking forward to seem very fragile to different kinds of moral panics or [01:45:08] political economy problems. I don’t actually agree that [01:45:12] much regarding the developed world. I feel like in the developed world, [01:45:17] markets function pretty well. When there’s a lot of money to [01:45:23] be made on something and it’s clearly the best available alternative, it’s actually hard for [01:45:27] the regulatory system to stop it. We’re seeing that in AI itself. [01:45:33] A thing I’ve been trying to fight for is export controls on chips to China. [01:45:38] That’s in the national security interest of the US. [01:45:42] That’s squarely within the policy beliefs of almost everyone in Congress of both parties. [01:45:52] The case is very clear. The counterarguments against it, I’ll politely call them fishy. [01:45:59] Yet it doesn’t happen and we sell the chips because there’s so much money riding on it. [01:46:08] That money wants to be made. In that case, in my opinion, that’s a bad thing. [01:46:13] But it also applies when it’s a good thing. So if we’re talking about drugs and benefits of [01:46:23] the technology, I am not as worried about those benefits being hampered in the developed world. [01:46:30] I am a little worried about them going too slow. As I said, I do think we should work to speed [01:46:37] the approval process in the FDA. I do think we should fight against [01:46:41] these chatbot bills that you’re describing. Described individually, I’m against them. I [01:46:46] think they’re stupid. But I actually think the bigger worry is the developing world, where we [01:46:51] don’t have functioning markets and where we often can’t build on the technology that we’ve had. [01:46:58] I worry more that those folks will get left behind. [01:47:01] And I worry that even if the cures are developed, maybe there’s someone in rural [01:47:04] Mississippi who doesn’t get it as well. That’s a smaller version of the concern [01:47:10] we have in the developing world. So the things we’ve been doing [01:47:14] are working with philanthropists. We work with folks who deliver medicine and [01:47:26] health interventions to the developing world, to sub-Saharan Africa, India, Latin America, [01:47:34] and other developing parts of the world. That’s the thing I think that [01:47:39] won’t happen on its own. You mentioned export controls. [01:47:42] Why shouldn’t the US and China both have a “country of geniuses in a data center”? [01:47:48] Why won’t it happen or why shouldn’t it happen? Why shouldn’t it happen. [01:47:54] If this does happen, we could have a few situations. [01:48:02] If we have an offense-dominant situation, we could have a situation [01:48:05] like nuclear weapons, but more dangerous. Either side could easily destroy everything. [01:48:14] We could also have a world where it’s unstable. The nuclear equilibrium is [01:48:19] stable because it’s deterrence. But let’s say there was uncertainty about, [01:48:24] if the two AIs fought, which AI would win? That could create instability. You often have [01:48:30] conflict when the two sides have a different assessment of their likelihood of winning. [01:48:34] If one side is like, “Oh yeah, there’s a 90% chance I’ll win,” and the other side thinks [01:48:40] the same, then a fight is much more likely. They can’t both be right, [01:48:43] but they can both think that. But this seems like a fully general argument [01:48:46] against the diffusion of AI technology. That’s the implication of this world. [01:48:52] Let me just go on, because I think we will get diffusion eventually. [01:48:55] The other concern I have is that governments will oppress their own people with AI. [01:49:04] I’m worried about a world where you have a country in which there’s already a government that’s [01:49:16] building a high-tech authoritarian state. To be clear, this is about the government. [01:49:21] This is not about the people. We need to find a way for [01:49:24] people everywhere to benefit. My worry here is about governments. [01:49:30] My worry is if the world gets carved up into two pieces, one of those two pieces [01:49:33] could be authoritarian or totalitarian in a way that’s very difficult to displace. [01:49:39] Now, will governments eventually get powerful AI, and is there a risk of authoritarianism? [01:49:45] Yes. Will governments eventually get powerful AI, and is there a risk of [01:49:52] bad equilibria? Yes, I think both things. But the initial conditions matter. At some point, we’re [01:50:00] going to need to set up the rules of the road. I’m not saying that one country, either the United [01:50:05] States or a coalition of democracies—which I think would be a better setup, although it [01:50:09] requires more international cooperation than we currently seem to want to make—should just say, [01:50:19] “These are the rules of the road.” There’s going to be some negotiation. [01:50:22] The world is going to have to grapple with this. What I would like is for the democratic nations of [01:50:31] the world—those whose governments represent closer to pro-human values—are holding the [01:50:39] stronger hand and have more leverage when the rules of the road are set. [01:50:44] So I’m very concerned about that initial condition. [01:50:47] I was re-listening to the interview from three years ago, and one of the ways it [01:50:51] aged poorly is that I kept asking questions assuming there was going to be some key [01:50:55] fulcrum moment two to three years from now. In fact, being that far out, it just seems [01:51:00] like progress continues, AI improves, AI is more diffused, and people will use it for more things. [01:51:05] It seems like you’re imagining a world in the future where the countries get together, and [01:51:09] “Here’s the rules of the road, here’s the leverage we have, and here’s the leverage you have.” [01:51:13] But on the current trajectory, everybody will have more AI. [01:51:18] Some of that AI will be used by authoritarian countries. [01:51:20] Some of that within the authoritarian countries will be used by private [01:51:22] actors versus state actors. It’s not clear who will benefit more. [01:51:26] It’s always unpredictable to tell in advance. It seems like the internet privileged [01:51:30] authoritarian countries more than you would’ve expected. [01:51:33] Maybe AI will be the opposite way around. I want to better understand what [01:51:38] you’re imagining here. Just to be precise about it, [01:51:42] I think the exponential of the underlying technology will continue as it has before. [01:51:47] The models get smarter and smarter, even when they get to a “country of geniuses in a data center.” [01:51:53] I think you can continue to make the model smarter. [01:51:56] There’s a question of getting diminishing returns on their value in the world. [01:52:01] How much does it matter after you’ve already solved human biology? [01:52:07] At some point you can do harder, more abstruse math problems, but nothing after that matters. [01:52:12] Putting that aside, I do think the exponential will continue, but there will be certain [01:52:18] distinguished points on the exponential. Companies, individuals, and countries [01:52:24] will reach those points at different times. In “The Adolescence of Technology” I talk about: [01:52:31] Is a nuclear deterrent still stable in the world of AI? [01:52:38] I don’t know, but that’s an example of one thing we’ve taken for granted. [01:52:42] The technology could reach such a level that we can no longer be certain of it. [01:52:50] Think of others. There are points where if you reach a certain level, maybe you have offensive [01:52:57] cyber dominance, and every computer system is transparent to you after that unless the [01:53:04] other side has an equivalent defense. I don’t know what the critical moment [01:53:09] is or if there’s a single critical moment. But I think there will be either a critical [01:53:14] moment, a small number of critical moments, or some critical window where AI confers [01:53:22] some large advantage from the perspective of national security, and one country or [01:53:30] coalition has reached it before others. I’m not advocating that they just say, [01:53:36] “Okay, we’re in charge now.” That’s not how I think about it. [01:53:42] The other side is always catching up. There are extreme actions you’re not [01:53:45] willing to take, and it’s not right to take complete control anyway. [01:53:52] But at the point that happens, people are going to understand that the world has changed. [01:53:58] There’s going to be some negotiation, implicit or explicit, about what the [01:54:05] post-AI world order looks like. My interest is in making that [01:54:14] negotiation be one in which classical liberal democracy has a strong hand. [01:54:24] I want to understand what that better means, because you say in the essay, [01:54:27] “Autocracy is simply not a form of government that people can accept in the post-powerful AI age.” [01:54:33] That sounds like you’re saying the CCP as an institution cannot exist after we get AGI. [01:54:41] That seems like a very strong demand, and it seems to imply a world where the leading lab [01:54:47] or the leading country will be able to—and by that language, should get to—determine [01:54:54] how the world is governed or what kinds of governments are, and are not, allowed. [01:55:02] I believe that paragraph said something like, “You could take it even further and say X.” [01:55:13] I wasn’t necessarily endorsing that view. I was saying, [01:55:18] “Here’s a weaker thing that I believe. We have to worry a lot about authoritarians and [01:55:24] we should try to check them and limit their power. You could take this much further and have a more [01:55:30] interventionist view that says authoritarian countries with AI are these self-fulfilling [01:55:38] cycles that are very hard to displace, so you just need to get rid of them from the beginning.” [01:55:43] That has exactly all the problems you say. If you were to make a commitment to [01:55:49] overthrowing every authoritarian country, they would take a bunch of actions now [01:55:53] that could lead to instability. That just may not be possible. [01:56:02] But the point I was making that I do endorse is that it is quite possible that… [01:56:09] Today, the view, my view, in most of the Western world is that democracy is a better form of [01:56:16] government than authoritarianism. But if a country’s authoritarian, [01:56:21] we don’t react the way we’d react if they committed a genocide or something. [01:56:27] I guess what I’m saying is I’m a little worried that in the age of AGI, authoritarianism will [01:56:32] have a different meaning. It will be a graver thing. [01:56:35] We have to decide one way or another how to deal with that. [01:56:39] The interventionist view is one possible view. I was exploring such views. It may end up being the [01:56:47] right view, or it may end up being too extreme. But I do have hope. One piece of hope I have is [01:56:55] that we have seen that as new technologies are invented, forms of government become obsolete. [01:57:04] I mentioned this in “Adolescence of Technology”, where I said feudalism [01:57:10] was basically a form of government, and when we invented industrialization, feudalism was no [01:57:18] longer sustainable. It no longer made sense. Why is that hope? Couldn’t that imply that [01:57:23] democracy is no longer going to be a competitive system? [01:57:26] Right, it could go either way. But these problems with [01:57:38] authoritarianism get deeper. I wonder if that’s an indicator of [01:57:44] other problems that authoritarianism will have. In other words, because authoritarianism becomes [01:57:52] worse, people are more afraid of it. They work harder to stop it. [01:57:59] You have to think in terms of total equilibrium. I just wonder if it will motivate new ways [01:58:07] of thinking about how to preserve and protect freedom with the new technology. [01:58:13] Even more optimistically, will it lead to a collective reckoning and a more emphatic [01:58:22] realization of how important some of the things we take as individual rights are? [01:58:27] A more emphatic realization that we really can’t give these away. [01:58:32] We’ve seen there’s no other way to live that actually works. [01:58:39] I am actually hopeful that—it sounds too idealistic, but I believe it could be the [01:58:46] case—dictatorships become morally obsolete. They become morally unworkable forms of [01:58:52] government and the crisis that that creates is sufficient to force us to find another way. [01:59:03] I think there is genuinely a tough question here which I’m not sure how you resolve. [01:59:07] We’ve had to come out one way or another on it through history. [01:59:11] With China in the ’70s and ’80s, we decided that even though it’s an [01:59:15] authoritarian system, we will engage with it. I think in retrospect that was the right call, [01:59:18] because it’s a state authoritarian system but a billion-plus people are much wealthier and [01:59:23] better off than they would’ve otherwise been. It’s not clear that it would’ve stopped being [01:59:27] an authoritarian country otherwise. You can just look at North Korea [01:59:30] as an example of that. I don’t know if it takes [01:59:34] that much intelligence to remain an authoritarian country that continues to coalesce its own power. [01:59:40] You can imagine a North Korea with an AI that’s much worse than everybody else’s, [01:59:44] but still enough to keep power. In general, it seems like we should just [01:59:50] have this attitude that the benefits of AI—in the form of all these empowerments [01:59:54] of humanity and health—will be big. Historically, we have decided it’s good [02:00:00] to spread the benefits of technology widely, even to people whose governments are authoritarian. [02:00:06] It is a tough question, how to think about it with AI, but historically we have said, “yes, [02:00:10] this is a positive-sum world, and it’s still worth diffusing the technology.” [02:00:15] There are a number of choices we have. Framing this as a government-to-government [02:00:20] decision in national security terms is one lens, but there are a lot of other lenses. [02:00:27] You could imagine a world where we produce all these cures to diseases. [02:00:32] The cures are fine to sell to authoritarian countries, but the data centers just aren’t. [02:00:38] The chips and the data centers aren’t, and the AI industry itself isn’t. [02:00:44] Another possibility I think folks should think about is this. [02:00:49] Could there be developments we can make—either that naturally happen as a result of AI, [02:00:55] or that we could make happen by building technology on AI—that [02:00:59] create an equilibrium where it becomes infeasible for authoritarian countries [02:01:05] to deny their people private use of the benefits of the technology? [02:01:12] Are there equilibria where we can give everyone in an authoritarian country their own AI model that [02:01:19] defends them from surveillance and there isn’t a way for the authoritarian country to crack [02:01:24] down on this while retaining power? I don’t know. That sounds to me like if that went far enough, [02:01:29] it would be a reason why authoritarian countries would disintegrate from the inside. [02:01:35] But maybe there’s a middle world where there’s an equilibrium where, if they want to hold on [02:01:39] to power, the authoritarians can’t deny individualized access to the technology. [02:01:45] But I actually do have a hope for the more radical version. [02:01:50] Is it possible that the technology might inherently have properties—or [02:01:54] that by building on it in certain ways we could create properties—that have this [02:02:01] dissolving effect on authoritarian structures? Now, we hoped originally—think back to the [02:02:07] beginning of the Obama administration—that social media and the internet would have [02:02:13] that property, and it turns out not to. But what if we could try again with the [02:02:20] knowledge of how many things could go wrong, and that this is a different technology? [02:02:23] I don’t know if it would work, but it’s worth a try. [02:02:26] It’s just very unpredictable. There are first principles reasons why [02:02:30] authoritarianism might be privileged. It’s all very unpredictable. We just [02:02:35] have to recognize the problem and come up with 10 things we can try, try those, [02:02:40] and then assess which ones are working, if any. Then try new ones if the old ones aren’t working. [02:02:46] But I guess that nets out to today, as you say, that we will not sell data centers, [02:02:51] or chips, and the ability to make chips to China. So in some sense, you are denying… There would be [02:02:58] some benefits to the Chinese economy, Chinese people, et cetera, because we’re doing that. [02:03:02] Then there’d also be benefits to the American economy because it’s a positive-sum world. [02:03:06] We could trade. They could have their country’s data centers doing one thing. [02:03:08] We could have ours doing another. Already, you’re saying it’s not worth that [02:03:14] positive-sum stipend to empower those countries? What I would say is that we are about to be [02:03:22] in a world where growth and economic value will come very easily if we’re [02:03:27] able to build these powerful AI models. What will not come easily is distribution [02:03:35] of benefits, distribution of wealth, political freedom. [02:03:40] These are the things that are going to be hard to achieve. [02:03:43] So when I think about policy, I think that the technology and the market will deliver all the [02:03:50] fundamental benefits, this is my fundamental belief, almost faster than we can take them. [02:03:55] These questions about distribution and political freedom and rights are the ones that will actually [02:04:02] matter and that policy should focus on. Speaking of distribution, as you were [02:04:06] mentioning, we have developing countries. In many cases, catch-up growth has been [02:04:12] weaker than we would have hoped for. But when catch-up growth does happen, [02:04:15] it’s fundamentally because they have underutilized labor. [02:04:18] We can bring the capital and know-how from developed countries to these countries, [02:04:21] and then they can grow quite rapidly. Obviously, in a world where labor is no [02:04:26] longer the constraining factor, this mechanism no longer works. [02:04:30] So is the hope basically to rely on philanthropy from [02:04:33] the people or countries who immediately get wealthy from AI? What is the hope? [02:04:38] Philanthropy should obviously play some role, as it has in the past. [02:04:44] But I think growth is always better and stronger if we can make it endogenous. [02:04:50] What are the relevant industries in an AI-driven world? [02:04:58] I said we shouldn’t build data centers in China, but there’s no reason we shouldn’t [02:05:00] build data centers in Africa. In fact, I think it’d be [02:05:04] great to build data centers in Africa. As long as they’re not owned by China, [02:05:08] we should build data centers in Africa. I think that’s a great thing to do. [02:05:16] There’s no reason we can’t build a pharmaceutical industry that’s AI-driven. [02:05:22] If AI is accelerating drug discovery, then there will be a bunch of biotech startups. [02:05:28] Let’s make sure some of those happen in the developing world. [02:05:31] Certainly, during the transition—we can talk about the point where humans have no [02:05:34] role—humans will still have some role in starting up these companies and supervising the AI models. [02:05:41] So let’s make sure some of those humans are in the developing world [02:05:44] so that fast growth can happen there as well. You guys recently announced that Claude is going [02:05:48] to have a constitution that’s aligned to a set of values, and not necessarily just to the end user. [02:05:53] There’s a world I can imagine where if it is aligned to the end user, [02:05:56] it preserves the balance of power we have in the world today because everybody gets to have their [02:05:59] own AI that’s advocating for them. The ratio of bad actors to [02:06:03] good actors stays constant. It seems to work out for our world today. [02:06:07] Why is it better not to do that, but to have a specific set of values that the [02:06:12] AI should carry forward? I’m not sure I’d quite [02:06:16] draw the distinction in that way. There may be two relevant distinctions here. [02:06:22] I think you’re talking about a mix of the two. One is, should we give the model a set of [02:06:27] instructions about “do this” versus “don’t do this”? [02:06:31] The other is, should we give the model a set of principles for how to act? [02:06:44] It’s kind of purely a practical and empirical thing that we’ve observed. [02:06:48] By teaching the model principles, getting it to learn from principles, [02:06:52] its behavior is more consistent, it’s easier to cover edge cases, and the model is more [02:06:58] likely to do what people want it to do. In other words, if you give it a list of [02:07:09] rules—“don’t tell people how to hot-wire a car, don’t speak in Korean”—it doesn’t [02:07:10] really understand the rules, and it’s hard to generalize from them. [02:07:15] It’s just a list of do’s and don’t’s. Whereas if you give it principles—it [02:07:21] has some hard guardrails like “Don’t make biological weapons” but—overall you’re [02:07:25] trying to understand what it should be aiming to do, how it should be aiming to operate. [02:07:31] So just from a practical perspective, that turns out to be a more effective way to train the model. [02:07:35] That’s the rules versus principles trade-off. Then there’s another thing you’re talking about, [02:07:42] which is the corrigibility versus intrinsic motivation trade-off. [02:07:51] How much should the model be a kind of “skin suit” where it just directly [02:08:02] follows the instructions given to it by whoever is giving those instructions, [02:08:06] versus how much should the model have an inherent set of values and go off and do things on its own? [02:08:14] There I would actually say everything about the model is closer to the direction that [02:08:21] it should mostly do what people want. It should mostly follow instructions. [02:08:24] We’re not trying to build something that goes off and runs the world on its own. [02:08:29] We’re actually pretty far on the corrigible side. Now, what we do say is there are certain [02:08:34] things that the model won’t do. I think we say it in various ways in the [02:08:40] constitution, that under normal circumstances, if someone asks the model to do a task, it should do [02:08:45] that task. That should be the default. But if you’ve asked it to do something dangerous, or [02:08:54] to harm someone else, then the model is unwilling to do that. [02:09:01] So I actually think of it as a mostly corrigible model that has some limits, [02:09:07] but those limits are based on principles. Then the fundamental question is, [02:09:12] how are those principles determined? This is not a special question for Anthropic. [02:09:15] This would be a question for any AI company. But because you have been the ones to actually [02:09:22] write down the principles, I get to ask you this question. [02:09:25] Normally, a constitution is written down, set in stone, and there’s a process of [02:09:29] updating it and changing it and so forth. In this case, it seems like a document [02:09:34] that people at Anthropic write, that can be changed at any time, [02:09:37] that guides the behavior of systems that are going to be the basis of a lot of economic activity. [02:09:45] How do you think about how those principles should be set? [02:09:50] I think there are maybe three sizes of loop here, three ways to iterate. [02:09:58] One is we iterate within Anthropic. We train the model, we’re not happy with it, [02:10:01] and we change the constitution. I think that’s good to do. [02:10:06] Putting out public updates to the constitution every once in a while [02:10:10] is good because people can comment on it. The second level of loop is different companies [02:10:16] having different constitutions. I think it’s useful. Anthropic puts out a constitution, [02:10:21] Gemini puts out a constitution, and other companies put out a constitution. [02:10:28] People can look at them and compare. Outside observers can critique and say, [02:10:34] “I like this thing from this constitution and this thing from that constitution.” [02:10:40] That creates a soft incentive and feedback for all the companies to [02:10:45] take the best of each element and improve. Then I think there’s a third loop, which is [02:10:50] society beyond the AI companies and beyond just those who comment without hard power. [02:10:59] There we’ve done some experiments. A couple years ago, we did an experiment with the Collective [02:11:04] Intelligence Project to basically poll people and ask them what should be in our AI constitution. [02:11:15] At the time, we incorporated some of those changes. [02:11:17] So you could imagine doing something like that with the new approach we’ve [02:11:19] taken to the constitution. It’s a little harder because [02:11:23] it was an easier approach to take when the constitution was a list of dos and don’ts. [02:11:29] At the level of principles, it has to have a certain amount of coherence. [02:11:32] But you could still imagine getting views from a wide variety of people. [02:11:37] You could also imagine—and this is a crazy idea, but this whole [02:11:42] interview is about crazy ideas—systems of representative government having input. [02:11:52] I wouldn’t do this today because the legislative process is so slow. [02:11:55] This is exactly why I think we should be careful about the legislative process and AI regulation. [02:12:00] But there’s no reason you couldn’t, in principle, say, “All AI models have to have a constitution [02:12:06] that starts with these things, and then you can append other things after it, but there has to [02:12:13] be this special section that takes precedence.” I wouldn’t do that. That’s too rigid and sounds [02:12:22] overly prescriptive in a way that I think overly aggressive legislation is. [02:12:26] But that is a thing you could try to do. Is there some much less heavy-handed [02:12:32] version of that? Maybe. I really like control loop two. [02:12:37] Obviously, this is not how constitutions of actual governments do or should work. [02:12:42] There’s not this vague sense in which the Supreme Court will feel out how people [02:12:46] are feeling—what are the vibes—and update the constitution accordingly. [02:12:50] With actual governments, there’s a more formal, procedural process. [02:12:55] But you have a vision of competition between constitutions, which is actually very reminiscent [02:13:01] of how some libertarian charter cities people used to talk, about what an archipelago of different [02:13:07] kinds of governments would look like. There would be selection among them of [02:13:10] who could operate the most effectively and where people would be the happiest. [02:13:15] In a sense, you’re recreating that vision of a utopia of archipelagos. [02:13:23] I think that vision has things to recommend it and things that will go wrong with it. [02:13:31] It’s an interesting, in some ways compelling, vision, but things will [02:13:34] go wrong that you hadn’t imagined. So I like loop two as well, [02:13:40] but I feel like the whole thing has got to be some mix of loops one, two, and three, [02:13:46] and it’s a matter of the proportions. I think that’s gotta be the answer. [02:13:53] When somebody eventually writes the equivalent of The Making of the Atomic Bomb for this era, [02:13:58] what is the thing that will be hardest to glean from the historical record that [02:14:02] they’re most likely to miss? I think a few things. One is, [02:14:06] at every moment of this exponential, the extent to which the world outside it didn’t understand it. [02:14:12] This is a bias that’s often present in history. Anything that actually happened looks [02:14:17] inevitable in retrospect. When people look back, it will [02:14:24] be hard for them to put themselves in the place of people who were actually making a bet on this [02:14:32] thing to happen that wasn’t inevitable, that we had these arguments like the arguments I make for [02:14:38] scaling or that continual learning will be solved. Some of us internally put a high probability [02:14:48] on this happening, but there’s a world outside us that’s not acting on that at all. [02:14:58] I think the weirdness of it, unfortunately the insularity of it… [02:15:07] If we’re one year or two years away from it happening, [02:15:10] the average person on the street has no idea. That’s one of the things I’m trying to change with [02:15:14] the memos, with talking to policymakers. I don’t know but I think [02:15:19] that’s just a crazy thing. Finally, I would say—and this [02:15:27] probably applies to almost all historical moments of crisis—how absolutely fast it was happening, [02:15:33] how everything was happening all at once. Decisions that you might think were [02:15:39] carefully calculated, well actually you have to make that decision, [02:15:42] and then you have to make 30 other decisions on the same day because it’s all happening so fast. [02:15:47] You don’t even know which decisions are going to turn out to be consequential. [02:15:52] One of my worries—although it’s also an insight into what’s happening—is that some [02:16:00] very critical decision will be some decision where someone just comes into my office and [02:16:05] is like, “Dario, you have two minutes. Should we do thing A or thing B on this?” [02:16:14] Someone gives me this random half-page memo and asks, “Should we do A or B?” I’m like, “I [02:16:20] don’t know. I have to eat lunch. Let’s do B.” That ends up being the most consequential thing ever. [02:16:26] So final question. There aren’t tech CEOs who are usually writing 50-page memos every few months. [02:16:35] It seems like you have managed to build a role for yourself and a company around [02:16:40] you which is compatible with this more intellectual-type role of CEO. [02:16:47] I want to understand how you construct that. How does that work? Do you just go away for [02:16:53] a couple of weeks and then you tell your company, “This is the memo. Here’s what [02:16:56] we’re doing”? It’s also reported that you write a bunch of these internally. [02:16:59] For this particular one, I wrote it over winter break. [02:17:04] I was having a hard time finding the time to actually write it. [02:17:08] But I think about this in a broader way. I think it relates to the culture of the company. [02:17:13] I probably spend a third, maybe 40%, of my time making sure the culture of Anthropic is good. [02:17:19] As Anthropic has gotten larger, it’s gotten harder to get directly involved in the training [02:17:26] of the models, the launch of the models, the building of the products. It’s 2,500 [02:17:30] people. I have certain instincts, but it’s very difficult to get involved in every single detail. [02:17:41] I try as much as possible, but one thing that’s very leveraged is making sure Anthropic is a good [02:17:46] place to work, people like working there, everyone thinks of themselves as team members, and everyone [02:17:51] works together instead of against each other. We’ve seen as some of the other AI companies [02:17:57] have grown—without naming any names—we’re starting to see decoherence and people fighting each other. [02:18:03] I would argue there was even a lot of that from the beginning, but it’s gotten worse. [02:18:08] I think we’ve done an extraordinarily good job, even if not perfect, of holding the [02:18:14] company together, making everyone feel the mission, that we’re sincere about the mission, [02:18:19] and that everyone has faith that everyone else there is working for the right reason. [02:18:23] That we’re a team, that people aren’t trying to get ahead at each other’s expense or [02:18:28] backstab each other, which again, I think happens a lot at some of the other places. [02:18:33] How do you make that the case? It’s a lot of things. It’s me, [02:18:36] it’s Daniela, who runs the company day to day, it’s the co-founders, [02:18:41] it’s the other people we hire, it’s the environment we try to create. [02:18:44] But I think an important thing in the culture is that the other leaders as well, but especially me, [02:18:53] have to articulate what the company is about, why it’s doing what it’s doing, [02:18:58] what its strategy is, what its values are, what its mission is, and what it stands for. [02:19:06] When you get to 2,500 people, you can’t do that person by person. [02:19:09] You have to write, or you have to speak to the whole company. [02:19:12] This is why I get up in front of the whole company every two weeks and speak for an hour. [02:19:18] I wouldn’t say I write essays internally. I do two things. One, I write this thing [02:19:22] called a DVQ, Dario Vision Quest. I wasn’t the one who named it that. [02:19:27] That’s the name it received, and it’s one of these names that I tried to fight because it made it [02:19:32] sound like I was going off and smoking peyote or something. But the name just stuck. So I get up [02:19:38] in front of the company every two weeks. I have a three or four-page document, [02:19:43] and I just talk through three or four different topics about what’s going on internally, [02:19:49] the models we’re producing, the products, the outside industry, the world as a whole [02:19:54] as it relates to AI and geopolitically in general. Just some mix of that. I go [02:19:59] through very honestly and I say, “This is what I’m thinking, and this is what Anthropic leadership [02:20:06] is thinking,” and then I answer questions. That direct connection has a lot of value that [02:20:13] is hard to achieve when you’re passing things down the chain six levels deep. [02:20:19] A large fraction of the company comes to attend, either in person or virtually. [02:20:27] It really means that you can communicate a lot. The other thing I do is I have a channel in [02:20:32] Slack where I just write a bunch of things and comment a lot. [02:20:36] Often that’s in response to things I’m seeing at the company or questions people ask. [02:20:44] We do internal surveys and there are things people are concerned about, and so I’ll write them up. [02:20:50] I’m just very honest about these things. I just say them very directly. [02:20:56] The point is to get a reputation of telling the company the truth about what’s happening, to call [02:21:01] things what they are, to acknowledge problems, to avoid the sort of corpo speak, the kind of [02:21:07] defensive communication that often is necessary in public because the world is very large and full of [02:21:14] people who are interpreting things in bad faith. But if you have a company of people who you trust, [02:21:21] and we try to hire people that we trust, then you can really just be entirely unfiltered. [02:21:31] I think that’s an enormous strength of the company. [02:21:33] It makes it a better place to work, it makes people more than the sum of their parts, [02:21:38] and increases the likelihood that we accomplish the mission because everyone is on the same page [02:21:41] about the mission, and everyone is debating and discussing how best to accomplish the mission. [02:21:46] Well, in lieu of an external Dario Vision Quest, we have this interview. [02:21:50] This interview is a little like that. This has been fun, Dario. Thanks for doing it. [02:21:54] Thank you, Dwarkesh.

42 AI