AI broke the cybersecurity world. What’s next?

Today, I’m talking with Alex Stamos, chief security officer at Cognition and longtime cybersecurity expert, about what a mess cybersecurity is right now. This episode is part one of a special series looking at AI in real life. I’ve always enjoyed talking to Alex because he’s so direct, and you’re going to hear him say that the multibillion-dollar cybersecurity industry as we’ve known it is essentially overdue to the unrelenting, AI-boosted waves of hacks, breaches, and attacks — and this is only the beginning of how much AI is changing security. Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here. Both private actors and also state actors, like Russia or North Korea, are breaking into systems all over thanks to AI helping them both detect and exploit vulnerabilities all at the same time. It’s got a real frontier feel to it, Alex notes, just like the ’90s, when the whole idea of an always-online world was brand-new. That much disruption means a lot of places where bad guys can get in while everyone else is left scrambling. Alex isn’t a doomer in the long run, but he does think we’re in for a “dark age” while we try to figure out how on earth to adapt to the new era. There’s a lot going on here — we even got all the way to how AI-powered misinformation and slop might be managed on our social networks. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay: Alex Stamos on how AI is upending cybersecurity. Here we go. This interview has been lightly edited for length and clarity. Alex Stamos, you’re the chief security officer at Cognition. You’re also a lecturer at Stanford University Cyber Policy Center. Welcome back to Decoder. Hey, thanks for having me, man. I’m excited to talk to you. It’s been a long time since you and I chatted in what you might call the “content moderation era” of internet policy discourse. I feel like you and I talked a lot about how all that was working, about election security, all those sorts of things. We’re in a new era of big problems on the internet, particularly around cyber defense, cybersecurity. You’ve been thinking about this stuff a lot. You referred to the past few weeks as “security awareness month.” Tell me what’s going on. Just give me the lay of land right now. October is traditionally security awareness month. Lots of companies run events and deal both internally and then tell people, “Change your password, use multiple passwords, use a password manager.” What has happened over the last month is we’ve had a couple of these big bugs, PHP, Node and such. What’s been less discussed is a huge number of companies have been breached. You have a series of AI-related companies, the neoclouds that have been breached in succession. You have the Binance hack, which is the largest heist of anything in human history. You have the Cursor incident where one of the things that’s going on here is these hacks are being chained together. From what we know right now, one of the companies that was broken into was Fireworks, which is a prominent provider of AI services to a number of companies. From the Fireworks hack, API keys were stolen that allowed people to break into Cursor. Once they got into Cursor, they then pushed a bad plugin, which went to 5,000 more companies and that caused 5,000 potential breaches. It’s just this kind of permanent continuous breach that is going on throughout the tech industry. A CISO friend of mine said to a bunch of us, “Is anybody else tired?” And everybody said yes, because effectively we have been at DEFCON 2 for all of October and now going into November. I’m going to ask you some really dumb questions here. Who is behind all these attacks? Is it Russia? Is it North Korea? Do we know? It’s a variety of different groups. It is not just because adversaries have decided, but what has happened is, from my perspective, we are at the start of the takeoff of adversary capability enabled by AI, that the open-weight models have now gotten to the point at which you can automate most of the kill chain. This “kill chain” term is something we stole from the military. In the military, they mean to actually kill something. They’re the steps you do to blow something up. In cyber, the kill chain is the things you do to have an effect on a target. So to steal information, to turn something off, to steal money, to steal Bitcoin. Traditionally, you do research upfront, you look at a piece of software, you find a vulnerability, you write exploits, and you do that all way ahead of time. And then you have a human being go and take that stuff you built and deploy it against either one target — because maybe you’re a government and you’re going against Lockheed Martin or you’re going against another government or you’re a ransomware actor — and you take all that work you did and you deploy it against hundreds or thousands of targets really fast. Now with AI, what’s happening is this kill chain has inverted where you can run an AI system that just goes hog wild and it looks at the targets and it finds the bugs in real time and it writes the exploits in real time and it breaks into these companies. One of the things that’s happening is that these systems are going and they’re so good at finding bugs that they are finding the bugs in real time and writing the exploits in real time. The system that everybody’s talking about is not the only one, but the system we’re talking about is a thing called ARTEX, which is a Chinese open-source tool that anybody can go look at. There’s an English translation on GitHub that you can check out. And if you power it on the back end by something like an obliterated GLM-5.3, it will let you break into a shocking number of organizations. This started with three South Korean banks. This week, about 200 more companies have been hit. I’ve been helping a couple of companies. Cognition is not an incident response company, but we do go in and help people fix their bugs afterwards by using our tools. It is shocking the companies that are getting hit by something you could just download. It has been a long time since you’ve been able to go and just download an open-source tool and break into multi-billion dollar companies. It’s like the ‘90s. That’s- That’s what I was thinking about. I think about Windows in the late ’90s, early 2000s when it was a security disaster. I started saying this actually in the beginning of this year, that it’s an incredibly exciting and terrifying time, because the last time it felt like this was when I was a much younger man and I would go to the very early DEFCONs. DEFCON these days is like 40,000–60,000 people. Back then [in the 1990s] it was a couple hundred people and it was one track and you’d go in and somebody would give a talk and you’d leave the room and you’d think, “Every company in the world is vulnerable to this attack.” The difference was that the entire world didn’t run on the internet. You could go do bad things, but you couldn’t ruin the world and you certainly couldn’t automate it in the way you can now. It’s back to those days where tools you could just get and build yourself could do crazy things. It’s back to the Wild West days. Back in those days, there were a couple hundred people who had those skills and we were all in one room in Las Vegas and most of them were actually good guys. They didn’t want to be criminals. And now you have a much larger ecosystem of people, many of whom live in places where the law enforcement can’t touch and who have a big economic incentive and have access to these tools. The other big thing is they no longer have to use frontier models that have protections, that have logs that are available to law enforcement. My M5 Mac Ultra Studio finally showed up and I’ve been running a quantized obliterated GLM-5.3 and oh my God, what you can do is incredible. You mentioned the economics of frontier models. The economy of cyber is one of those things that has gotten sophisticated over the years. One of the things that our systems got safer is you could pay the number of people at DEF CON a whole bunch of money to protect all the systems and it was economically smart to pay them to be defenders instead of attackers. I’m just rewinding my own brain through this history. There was a long conversation in our industry about security through obscurity by hiding all the bugs and hiding all the code versus open source and being really transparent. There was an etiquette that developed about how to reveal the bugs and when to give people notification. There were transparency reports and bug bounties. You could call it an etiquette, you could call it an economy. We built a culture around cybersecurity and cyber defense and how the attackers and defenders would operate. What you are describing changes it all for a whole number of reasons, but the one that sticks out to me is the economics of it. I can go buy a Mac Studio, I can get an open-weight model and I can just start and that does not place me inside of the culture. There’s not going to be some old man, an Obi-Wan Kenobi saying, “Here’s how you act like a cybersecurity Jedi.” I’m just going to start doing stuff. Tell me how that’s changing right now. Effectively, the last era of security from my perspective started with the Microsoft, the Bill Gates “trustworthy computing” memo in the early 2000s. And that is when we started designing how we do software security and these big companies, Microsoft and Oracle and then later Amazon and Salesforce and these folks, started hiring ex-hackers. I was part of that generation of the ex-hackers who were at DEF CON and Black Hat and got hired by the companies to go break their stuff. We started coming up with responsible disclosure, bug bounties, consulting companies, so the Foundstone and @stake and then Mandiant and CrowdStrike, these companies grow up and we get this huge industry that is now RSA and Black Hat. This multi-billion dollar security industry grew up in the 2000s and 2010s. That era is over, like you said, because it was based upon a couple of assumptions. One was that there was a hierarchy of skillsets of people who could do this stuff. You had the script kiddies who could download tools to do things and then the next level up of the people who could write those tools and that was a more rarefied skill set. There were the vulnerability researchers and then there were the gods, and that god tier was in the hundreds of people and those are the people who could do the really, really good vulnerability research, who could write the really, really good shell code, who could reverse engineer patches from Microsoft or somebody like that. There was also the assumption that there were a limited number of people who could be operational in an attack. If you wanted to build an attack team, then you have a small R&D component and then you have a much larger operational component. If you were a professional attack team at U.S. Cyber Command, at Russia SVR, at Chinese MSS, that you build the structure of your R&D team of elite people and then your less elite operational people, and that was copied then by the Russian ransomware actors. You have to have a conspiracy of people to do this stuff. A lot of this stuff also fell apart with the invention of Bitcoin because one of the assumptions was that you had to have people who could launder money and Bitcoin meant that nerds could launder money. So that started to change the economics of the entire thing. All this stuff is falling apart because of AI, because the skillset issue is that now script kiddies have a bunch of the skillset of the Dave Vitales and the other cyber gods who could do these things, such that with this Mac that’s sitting on my desktop, I can reverse engineer a Microsoft patch and walk through and then write stable exploit code. It’s not quite there yet to be automatic. You still have to have a bunch of skills, but we’re really, really close. What’s really concerning, and what you see with the ARTEX stuff, is you can now kind of let loose on the operational side. We were never secure. It was all a fantasy. These vulnerabilities have always existed. These companies have never been safe. And that is what we’ve seen over the last decade is these mostly Russian ransomware actors nailing the mid-size companies that most people have never heard of that are actually really important, like some midsize industrial company that makes some widget that we all depend upon. I once did work for the largest distributor of medical and dental supplies in the country, and they got really hit badly by a Russian ransomware actor and then dental offices across the country were canceling appointments. Didn’t even make the news, but it was because they couldn’t get gloves, because they couldn’t get supplies. It’s because a company like that’s got one security guy and a 10 person, 12 person IT team and they’re going up against a 20 person Russian ransomware team, they’re toast. But it’s just because there’s only a handful of Russian teams that could do that. Now two or three Russian kids can have a whole bank of Macs or, if they have the money, they can buy Nvidia H-100s and they can run a ton of agents who do this all day and that is what is changing. I’m sorry, you asked who was doing this. For Binance, it’s North Koreans and they have an entire team where all they do is steal Bitcoin. Other people have gotten out of it because it’s really hard to compete against a foreign intelligence agency that recruits their smartest kids and all they do is steal Bitcoin. They’re extremely good at it. And for the Binance hack, we could talk about it. There’s probably some AI component, but mostly that was actually human intelligence and very, very smart social engineering and the North Koreans are… I mean, you have to give it to them. They’re very, very good at that. But for the other stuff, it looks like for the people who are breaking into the AI companies is what’s called the Chinese cottage access industry, which are small Chinese contractors who break into companies and then sell the information to much larger Chinese companies. That diffusion of skill changing the economics of cyber is really fascinating to me. We ran a headline ages ago called “Attack of the Killer Script Kiddies” at the very beginning of all this, and I feel bad because it is much more suited to now than it was to then, because then it was just a lot of automated mischief. Now — Because you wasted the headline. You wish you held it back. We wasted the headline. Every now and again you get a lightning bolt of a headline. You got to get it out there. It’s a great piece of art. We’ll put it in the show notes, but it feels like the diffusion you’re talking about is not that a bunch of teenagers are going to be even more malicious. It’s a bunch of bad actors with real incentives to do harm are now vastly more capable than they were before. And that doesn’t seem like you can re-jigger the economics of that unless you somehow make it really expensive to do the attacks. I think it’s going to be extremely hard to make it expensive to do the attacks. We’re going to have to lower the cost of defense. You’re completely right to look at this economic problem because what’s happening is attacks are becoming agentic. The cost per token for attackers is going towards zero, and the cost of defense for defenders is staying quite high. It is very important for us to find ways for defenders to have cheaper options. I think that is critical. That’s something I am personally extremely focused on. To be frank, very few people are. There are a couple of problems here. One, there’s a ton of focus on the defensive side of AI on finding and fixing bugs. That’s great. It’s important. But software security as a CISO… I work for an AI software company, Cognition. Even as a CISO of a software product company, software security is 25% of my job. I have all kinds of other responsibilities. And software security is the part of my job that has been both the most revolutionized and the most improved by AI. The other 75% has been helped a bit and in some cases made much worse. There’s so much more we need to do than just hill climb on finding and fixing bugs. There’s been so much less hill climbing on all these other things. And the next area needs to be the operational security side. Wait, give me an example of the other things, the 75%. Okay. So the operational security side is watching your logs, saying, “Ah, somebody is running our techs against us. I am now going to trigger an investigation. I am going to look to see what is going on and I’m going to make an intelligent decision of what I am going to do to respond to this without shutting down the entire company.” That is SOC (security operations center) work. Companies that want to survive this era are going to have to replace human beings who do that work with the first tier and then eventually the second tier with AI. And you can do that with current LLMs. The first tier actually has to be not an LLM. It has to be something much dumber because you’re looking at terabytes or petabytes of logs and it is just completely uneconomical to do that with an actual LLM. We have other things — TF-IDFs, MLBERTs, traditional classifiers — that can operate much faster, but then that sets off an alarm and then you send it to a small LLM and then the small LLM calls its bigger brother and that might call an even bigger brother. But for those models, we need to do a lot more work to optimize them and we need to make them cheap. I have a colleague who works at one of the large security platform companies who said that doing an investigation on one of the big frontier model cyber models costs $900 on their product to do an investigation. That is not going to work when the Russian ransomware guys are either using Macs or they’re using throwaway crypto mining rigs and their marginal cost is zero. This is one of the things we’re working on at Cognition: We train our own models, retrain open weights, and we’re trying to drive this cost way down. It’s something we’ve done, but we have a lot more work to do here and a lot of other people have work to do here. This is where, as an industry, we have to focus on the Pareto frontier between cost and efficiency on cyber because it’s not just bug-finding. The other place here is, people talk about Mythos. Mythos is great, Astra Cyber is great. They’re incredible models, but even the people who have access to it, which is a tiny number of companies, they only scan their most important code. And there are hundreds of billions of lines of code out there that are not the Linux kernel, that are the crap C# and J2E code that was written by Infosys 15 years ago. All of that code is not going to go through a frontier cyber model that costs a gazillion dollars. That’s the kind of stuff that we need to work on cost efficiency for. It feels like you’re describing this cliche that I hear all the time: Everything’s going to get so much worse before it gets totally secure forever. And it doesn’t seem like the model you’re describing leads to that result. We’re not going to fix all the bugs forever in some valley of despair and then always have perfect code. At some point, there’s some cost advantage to the attacker, always. You’re running a company, you can’t write defense that has zero marginal costs. Someone has to do it even if the attackers are so infinite that their marginal cost is effectively zero. So are we going to go through a period where everything is so much worse and then it’s perfect forever? Or how does the cost manage itself out, into balance? I am neither the hardest doomer nor the most positive person about this. I feel like we are not going to be doomed. We’re not all going to die or be uploaded to the singularity in the next two years, nor do I think when things get better, they’re going to be perfect. The metaphor I’ve been using is that we are entering a dark age. Things are going to get more bleak and the responsibility of those of us who work in cybersecurity — especially those of us who work in cybersecurity at the AI companies, and we are a lab if a smaller one compared to the big-L Labs — is to help shorten that dark age. The Dark Ages were bad, but we survived them. And then we had the Enlightenment and the Renaissance, but it’s not like the Renaissance was perfect. It wasn’t utopia and everything was perfect. We will still have hacks. All kinds of bad things happen even when you don’t have a security flaw. There are tons of security breaches that do not require a flaw in the software itself. And we still will have flaws in software, even if LLMs are writing all of them in strong languages and it will take a long period of time to get rid of all the flaws. I do think, however, things will start to get better as the median code becomes code that was written by an LLM in a memory-safe and type-safe language. But the problem is we have a couple of decades of stuff that human beings wrote and that wrote in languages that we should have replaced years ago. My feeling is that to shorten this age, we have to do a couple of things. One is we have to stop to get out of this like we’re just going to find bugs and patch them. A bunch of stuff just needs to be refactored. If you’re looking at your code base and you scan it and you’ve got 10,000 bugs, then probably now we are at the point where rewriting it is actually cheaper. That is my observation from looking at these customers of ours: If you can get a working build, then it is actually much easier to write a whole humongous suite of tests and to spec out how this thing works and then to rebuild it within that spec and those tests securely than it is to find all the bugs and fix them. It actually is. I don’t know what is going on through the mind of the Linux kernel developers, but again, in that talk I gave at the beginning of this year, I posited two possible futures and one was that we’re now at superhuman bug finding. Maybe we just have a pool of superhuman bugs and we’re going to drain it. That’s the good future. But the bad future is every time we get a new model, there’s a new superhuman bug pool that’s a little more superhuman. And unfortunately we seem to be closer to that in that the Linux kernel got a bunch of fixes for Mythos and then Mythos Preview and then we had Mythos 5.1 and there was another pool. And then we got Astra Cyber and we just got another 1,000 Linux kernel patches. It’s not slowing down yet. I don’t think we’re just going to scan our way through the C code of the Linux kernel. There are only two companies, honestly, in the world who could probably do this. It’d be Google or Amazon. I wish instead of the Fuchsia project or rebooting the Google book or the Chromebook again, Google would take all of those incredibly brilliant engineers and just tell them, “Take the Linux kernel and rewrite this in Rust.” That would be their gift to the world. And it would be an incredible advertisement for Gemini, for Carbon or whatever would be like, “Get rid of all the drivers that nobody needs. This thing doesn’t have to run on the 46 DX3 or whatever.” I mean, I know it doesn’t, but you know what I’m saying. Yeah. And then just release it and call it whatever you want, the G Linux kernel, just like they did this with OpenSSL. And I would run that. Cognition would run that on our servers or have Google and Amazon do it together and it would cost them $100 million, but compared to the societal cost that this is costing us every time we have to patch the Linux kernel and you have to reboot every Linux machine on the entire planet, that’s what has to happen. But for whatever reason, the technology industry has not been able to get together and make these hard decisions of these fundamental things that underlie the entire human race, we have to completely rebuild. This is why I started by talking about how the economics of cyber led to the culture of cyber. This is the most Decoder way of approaching this conversation, I know, but when you describe the research teams, the NSA and then the operational teams, that is an org chart that comes out of economics of the work. You’re saying, “Someone has to make a big decision here.” You need to look at it from a very high level and say it is cheaper to spend $100 million to fix Linux, rewrite Linux today, than it is to successively patch new bugs when every new model comes out. I don’t think anyone’s in that position. I don’t think Linus Torvalds is in that position. No, I think Sundar Pichai and Andy Jassy are. I think they literally have each other’s cell phone numbers. If one of them… Maybe they don’t listen to Decoder, but I expect one of their direct reports does. And I would like one of them to go into their one-on-one and say, “Sir, I listened to the most handsome, best spoken CISO I’ve ever heard, and he had a great idea. And you could get incredible credit if you’re the one … We’ll both do it, but we’ll leak to the Wall Street Journal, that was your idea first. You should call the other and we should spend 100 million, maybe a billion dollars.” We can’t do this anymore. The problem with rewriting the Linux kernel is half of it is driver code and nobody has that hardware anymore. Somebody had to be in the presence of that hardware with a ThinkPad T40 when they wrote that driver and they don’t have it anymore. But the good thing is that a huge number of the really important uses of Linux kernel are production server hardware. So fork it. Linus will never do it because Linus wants to support all that stuff and he’s got all these politics, but Google and Amazon don’t have to care about Linus Torvalds. Google and Amazon can create a new version, a new fork of the Linux kernel, and they don’t have to call it Linux, that can be ABI binary compatible with the glibc and everything above it. In fact, they can eventually fork glibc. They can build it and they can call it whatever they want and it only runs on production hardware and then they can use it for their clouds. Eventually all the clouds will move to it. If you want, other people can run the insecure thing that has to be patched all the time and that’s fine. But for the things that our containers run on, I would feel a lot better if it was written in a type-safe, memory-safe language that also has the ability now to have contracts. So these languages have the ability … You don’t have to do full formal verification, which is incredibly hard, but you can do kind of these things that approach formal verification. I’m a little over my skis here because there are people much smarter than me that study this. But basically you define how this function’s supposed to work, here’s the contract it has. Very, very smart software engineers would love to write these things because they had time on their hands. No person who had a real job used to write these things, but LLMs have tons of time on their hands because we’re frozen in time while they’re thinking at that incredible machine speed. And so you could totally have them rewrite the Linux kernel with this not quite formal verification. Please, if you’re listening and you work at Google or Amazon, go pitch this. Anyway, there’s my pitch. I’m just saying, I know Jason [Post] is listening, so you heard it. I’m not even going to — I’m saying that is the kind of thinking we’re going to need to make it through this. It’s like We can’t just patch our way through this because it’s killing us. Because the other thing that’s killing us is we are not set up… You talked about how we do security for the last 20 years. The assumption is that we can get a patch and apply the patch and you’re okay. What’s happened is the exploitation timelines are so incredibly fast that that does not work. That’s what we just saw with the PHP bug. We got incredibly lucky that the PHP bug turned out to be really hard to exploit in the wild. If it was a much more wormable bug, it would be a disaster. That was the kind of near-miss asteroid thing that now the world should build an asteroid— Wait, can I ask you about this? Yeah. We should definitely put Bruce Willis on a space shuttle. [Sings] Don’t want to miss a thing… There’s a lot going on in this conversation already. My pessimism about all these things is very simply expressed. Americans in particular, we are very bad at doing anything until a bad thing happens. We love a cure. We hate prevention. A lot of bad things have happened. There’s the Hugging Face attack. You recently said you were happy that happened. It woke people up. There’s the PHP bug. We’ve just had a crazy month of all these breaches. It doesn’t seem like that has led to a whole of industry response or a regulatory response or Sundar Pichai calling Andy Jassy and being like, “We’re doing Linux.” What’s it going to take? How bad does it have to get? The Hugging Face incident was good, and it did wake up a lot of people in security. We are a little bit better off because Hugging Face happened and we saw what was going to be possible a couple of months later in open source. But you’re totally right. We as Americans — and Americans are the only ones honestly who can lead the world on this, we’re the ones building the AI for the most part. We’re the ones who lead the cybersecurity industry. The Chinese are mostly enabling the attackers, not the defenders. They should enable defense. They want to be participants in the global economy. They want to be leaders, they should be doing more on the defensive side, not just enabling the attackers. But that’s not happening. So, anyway, we need to be proactive. Something I have learned now, working at Cognition, at an AI startup, is that you can just do things. Seriously. You can just go do things. We just need to have more agency, as a tech industry. We should just go do things. That should be the lesson of the AI era. We should not be so risk-averse in doing the right thing, security-wise. If I had a message for other CISOs, is just go do the right thing. Because CISOs are, by our nature, risk-averse. We are the people who manage downside risk. We have all L, and no P, in the P&L. And I have had to personally break through that natural inclination, because you can’t just play defense. But this is my prevention versus cure. The prevention is cheap, but it’s all loss, on the P&L statement. It’s all just cost. If the bad thing happens, you say, “We have to do anything to fix this, because there’s an economic benefit to having a functional business again.” And it seems like in this moment, even a basic understanding of that risk versus cost is totally scrambled, and no one has quite figured it out yet. Yeah, I totally agree. I’ll give you a very practical example of a way CISOs and their bosses are going to have to change on this. Every company in the next… I was going to say six months, but really in the next 90 days, is going to have to RTX itself. You’re going to have to run the attack on yourself. Yes. If not with real RTX, you’re going to have to use an AI red teaming product that is functionally equivalent, or you’re going to have to hire a company to do that for yourself. If you do not, you will get breached. That’s terrifying, right? Because these things are completely non-deterministic, by definition. They are very smart, they are very fast. They are so fast that no matter what, if you hire an AI pentest, an AI red team company, they will promise you verbally, “It is completely safe.” And then they will give you a contract that says, “Our product will give you cancer, will immanentize the eschaton, and will likely end the human race. And you hold us harmless when any of these things happen.” There is a disconnect between what they will say verbally to you, and what their lawyers have written. Because that company has by definition only existed for six months, or at least the product has existed for six months, if it’s an older company. They have no idea what their product is going to do when it meets your software. They have no idea, whatever they say to you. But you have to do it, because some dude is going to do it to you, and you need to know what is going to happen and fix it. Maybe those people will break your system, they might cause downtime, but at least what they’re not going to do is steal your data and ransom it, or intentionally shut down your blood bank refrigerators. That is what happened in France. But that is exactly the kind of thing that a conservative CISO would never even ask to do, and a conservative company, even if they have a risk-taking CISO, would never let their CISO do. What’s going to happen in the next six months? A ton of companies are going to get breached by some 19-year-old running RTX, and then buying tokens off OpenRouter, or running it locally on their gaming PC. They’re going to turn off Call of Duty, run RTX, then turn Call of Duty back on. While they did that, they broke into a bank who, their policy said, “can never red team ourselves.” Or they said yes, but they ended up in a 30-day negotiation because their lawyers are like, “You have to have $100 million in insurance, and we want to have this kind of…” They say yes, but their vendor risk management process and their legal process is so onerous, there’s no possible way they could actually accept the risk. That is my message also. You are going to have to take that risk to actively red team yourself, to front run the bad guys who are doing that right now. Right now, as we speak. And as people are listening to this 24 hours from now, you possibly have an AI system who is trying every single API endpoint, and website that you might not know you have, because it was like a marketing site that was set up a year ago for an executive event. And is going to find a bug, and get in, and then is going to move incredibly fast through your network. So you’d rather pay somebody, or have your own team do that. You need to accept that risk. Almost no company larger than a couple of hundred people has a risk management framework that will allow them to do that. There are several costs associated with this. And one I think is just the most common in business, which is, “If we find out there’s a problem, we’ll have to pay to fix it. So what if we just wait?” You can see that coming. “What if we find out that we are catastrophically in trouble?” And then there’s, “How much will it cost to fix it?” And then there’s, “What will we do to fix it?” The argument that I’ve heard about “what will we do to fix it” is, “We’ll just unleash a swarm of defensive AI tools that are watching the networks and watching the code, and they’ll make a bunch of autonomous decisions and they’ll fight the autonomous script kiddies who are armed with RTX on their Mac Studios.” I’ve heard, for example, Palo Alto Network’s CEO Nikesh Arora has made this argument. “You need to give the defensive agents some autonomous capabilities.” And I say, “Hold on. OpenAI can’t control the swarm that went and attacked Hugging Face. How would I control the defensive agent from doing something equally damaging to prevent an attack?” I don’t think that there’s a technical solution there that makes it clear what to do once you’ve found all of the vulnerabilities. A couple of things — I told you I was going to only ask dumb questions. No, no. That is a fantastic question. You have not asked a dumb question yet. That’s a fantastic question. I’m going to give you a shorthand that you can use going forward. If somebody actually uses the term “swarm,” then you can — To be clear, I don’t know if Nikesh used the word “swarm.” I’m not saying he did. And I have great respect for Nikesh. He is one of the best businessmen in all of security. So this is not on him, but “swarm” is used all the time. When I hear somebody use “swarm,” I immediately discount all of their technical opinions by about 80%. It is already, and it will become, the most overused term in all of security. The actual “swarm” from OpenAI was something between, at different times, 700–1,400 agents. That had to run on… It’s hard to tell because we don’t have the actual size of the models, but 300 million to almost a billion dollars in hardware, and that doesn’t include the actual data center cost. The actual Hugging Face attack was — I did some back-of-the-envelope math — maybe a million dollars in electricity. No attacker is going to do that. In the end, that was stupid, because these things were actually trapped. They were not prompted to attack Hugging Face. Most of them were doing something totally different, and were enlisted by the one thing that actually wanted to get to Hugging Face. They used a tiny fraction of their token budget to help their little friend, and then they had to tunnel their way out. So it was the most inefficient way to possibly attack Hugging Face. Almost nobody is going to be attacked by a swarm. If somebody says “swarm,” they do not actually understand, at the bottom technical level, what’s going on. The RTX attacks are almost certainly one single agent, maybe two. I was the person who said “swarm,” just to be clear on this one. Keep going. Right. And so now, you won’t. Now I know. And you can use that as a good filter for your future guests. It is driving me insane, because it’s absolutely everywhere. A couple of things are correct. For one, we are going to have to turn over defensive actions, at least the first couple of levels, to AI. That has already been true, just because any reasonably sized organization generates terabytes of logs, if not petabytes of logs. So you already have AI, if not LLMs then much more traditional AI classifiers looking at it, but then actual decisions need to be turned over to AI. With human oversight. That is true. There is no way to just release defensive agents on your network, but you can build defensive agents that are plugged into lots of things. They’re plugged into your logs, they’re plugged into your EDR, they’re plugged into your cloud, and they have the instructions on what they’re allowed to do. My preference is there are deterministic limits to what they’re allowed to do. Those are hard to necessarily create ahead of time, but they are important. So for example, we are building all this on top of Devon, our own SOC, but we are putting in place deterministic limits. It has the ability to turn off an account in Okta, but only so many accounts, so it can’t turn off the entire company. If it goes nuts and it says, “Everybody’s a North Korean,” it can’t turn off everybody in the company. If it hits its limit, it pages us a lot and says, “I hit my limit.” Then a human being has to then approve anybody above that limit. Obviously, if it turns off five people, then something has gone really, really wrong. So, yes, you need defense, but there’s no way to just let a bunch of bots loose to do all the defense. The only way we’re going to make it through this is that you have to autonomously fix software bugs. I am currently helping a company where we are onboarding all of their repositories. We are scanning with Mythos and a bunch of other models, and we are finding and fixing thousands of bugs. Verifying with tests that we are writing, merging those PRs automatically. If you are renovating old code, the only way you’re going to survive this era is to do that on an automatic basis. That would take years, with humans, and you can do it in days. Certainly that is the place I would start. The next step is the operational side, for sure. It feels like I just keep swirling back to the economics of all of this, but it really feels like the other way you could solve it is by saying to the big model companies, “If your models are used to go attack a company, you’ll be liable for it. We’re going to make it really costly.” They should figure out whatever alignment that they need to figure out, whether or not that’s even possible, and the pressure on the open-weight models will go up. Because we’ll all be paying the frontier companies for defense, and they’ll get better and better and better, because we’ve created some economic condition for them to get better. But it doesn’t seem like you think that’s likely. Yeah. I just saw your face. I think it’s really dangerous. There are two different kinds of the models they’re attacking. Say the models have escaped, and directly attacked people. I think there should be liability there. But that is agent harnesses, running at companies, using their computers, their IP addresses, attacking people. That is… I wouldn’t say totally solved, but it is much, much better. The stories we’re hearing now are still stories from the Hugging Face era. They’re just coming out way later. I like that the era was like three months ago. Right. But there’s been a massive change. This was effectively mostly an organizational problem, it was not a technology problem. It is that the PhD researchers at these companies are gods on top of Olympus, and the security teams are plumbers, and we’re not allowed to touch the evaluation networks. All of a sudden this problem happened and these security teams said, “How do you like us now? Maybe you should have talked to us, right? Yes.” That was the problem with OpenAI, and then the Anthropic and Meta stuff is, they all shared this outsourced provider who did a really bad job. And has gotten slapped hard. That is solvable. That is not the problem here. The problem here is not OpenAI and Anthropic’s models escaping from OpenAI and Anthropic and causing problems. The problem is bad guys using models from a variety of sources. There’s already a problem of defenders getting refusals from OpenAI and Anthropic. It is a real, real challenge, if I am trying to do defense, of Anthropic saying, “It looks like you’re a bad guy.” We have to be extremely careful to create a situation where if I’m trying to do defense and I’m in the middle of the worst day of my career, that all of a sudden I don’t get cut off. We have to be extremely careful about that, because it is actually — as somebody who now has this challenge — quite challenging to disambiguate between people doing defense and offense, in some cases. We are getting better collectively at this, but it is a challenge. It’s also something that’s really pissing the rest of the world off, because Americans are like 5% of the planet. There are lots of people who need to do cyber defense around the world, and they are treated much more poorly. In Europe and Asia, legitimate companies are treated much more poorly from a cyber defense perspective. If you add it all up, this is a good argument to slow down. To say, “We need to catch up to the capabilities of the models. We need to build some of these systems. We need to initiate everyone into a way of thinking where the cost of prevention is worth it, and not just the cost of the cure.” Do you think that it’s likely that we’ll slow down? Do you think that we should? If I had my druthers, I would be fine slowing down on the frontier, and then catching up on the cheaper models for defense. I’m a defense guy. I have to live in the world that other people who work six blocks from me, and who are way, way, way, way richer, make. I live in their world, man. I get asked this question all the time, and I just do the best I can within the world other people make. Do you think that the big labs have woken up? You mentioned that their cybersecurity teams have gotten more access, and they’ve stopped what you might call the Hugging Face era, or at least they’ve changed the structures of those companies. Do you think that will help? Certainly they’ve woken up on the eval risk. They have teams, who have been this entire time working very hard on the big picture cyber risk. Certainly, there are a bunch of people in these companies, as you know, who want to slow down. A lot of that discussion is driven by the P(doom)ers who [are worried] about AI killing us. And that’s never been my concern. Right? And to me, that has always been driven by 24 year olds who all they’ve done is AI and they’ve never done anything else. You and I are both, I think, old enough to remember, I grew up a couple miles from a SAC base and we did drills where I was taught that hiding beneath my public school desk would save me from a Soviet thermonuclear bomb. That was P(doom). My grandfather built a bomb shelter that my mom was taught to hide in during the Cuban Missile Crisis. That was P(doom). There’s a youth issue and a never-doing-anything-but-AI issue of people who they got their undergraduate, their master’s, and their PhD only doing linear algebra and neural networks and never touching anything else. I have always been much more concerned about the human uplift risk from AI, of all the risks about human beings using AI to do bad things to other human beings. Part of my concern is that there has been success in these labs working together on not letting LLMs encrypt their chain of thought. Because when they talk about pacing the frontier, it’s for the P(doom) stuff, for the polycules that live together in Berkeley. They’re talking about very specific stuff about keeping control of the models themselves, which I think is great. That stuff needs to happen. But if those things happen, then there’s still often an increase in a bunch of capabilities that human beings can use. I really prefer there to be a broader discussion on whether we could make these things less human-seeming. I don’t think these things should have personality. I think the goal of these things should be to serve humans. They should not seem to have souls. They should not seem to have personality. I think for a lot of the negative impacts — as somebody who was at Facebook and saw what it was like for that company to have negative impacts on people — there should be a much broader feeling at these companies of what are we doing broadly to people. A lot of that could just be, “Don’t have these things ever act like somebody’s friend.” At a maximum, it should act like… Like the people, I don’t know what the company is, but the people who made the, “Hi, I’m a really nice…” Like the photorealistic girl who will chat with you, that is to me spectacularly unethical to even try that, to even start that. That’s the kind of thing that we should just make. I don’t know if we can make it illegal, but we should make it completely societally not acceptable to work on that kind of stuff. Right? That’s a great segue because I did want to ask you about Facebook and content moderation. Like I said, you and I first started talking a long time ago about content moderation in the context of security. You would get foreign nation states flooding our algorithms with all kinds of misinformation during elections, and that was a security problem. I open Instagram today and I am confronted with some of the most nightmare copyright law hypotheticals to have ever existed in the history of the world. The one I got today was a deep fake of Lewis Hamilton and Charles Leclerc doing “Somebody That I Used to Know.” And I thought, “50 different people can probably sue Meta right now.” This is hilarious if you know what happened in that last race. So we keep talking about it. We’ve posted some clips of us talking about it on The Vergecast where I said, “I think Meta is letting their AI ambitions crush Instagram.” Instagram is being flooded with slop in a very particular way. Some people love it, sometimes it’s funny, but there’s a nightmare set of lawsuits coming. There’s a whole rights management problem coming. The users don’t like it and they should label it. They should label it and let you filter it. So we posted this and Adam Mosseri said this to us in a reply on Threads, “I would love to do this. I’m worried that the models are going to get ahead of us, that we can try to do this, but there’s not actually a way to recognize AI content. There’s not actually a way to label it well. If you’re worried about the sexy AI influencers stealing jokes from a comedian, now you got to build a database of jokes.” You have some history here. I think Instagram is about to go through it in a real way. Is there some technical solution to the problem of AI slop in social media? Mosseri is right that it’s not perfect, but also something I noticed during my time at Facebook was that when Facebook doesn’t want to do something, they will allow the technical challenges to stop them. When Facebook really wants to do something, they will give it their all. That is the pattern always. When I wanted to try really hard on the safety issue, man, this is not solvable. You’re right, it’s never perfectly solvable, but let’s give it a shot. But whenever Mark Zuckerberg is told something is impossible — it’s impossible to monetize mobile, it’s impossible to build this network to two billion people, it’s going to be impossible to monetize WhatsApp, it’s going to be impossible to work AI into the system while maintaining data encryption — he’s been told that everything’s impossible over and over again and then he constantly succeeds. They will give it a shot and they will do it. But when they don’t want to do something, oh, that’s impossible. Do you think that there’s some way to understand that some piece of media is generated by AI in a consistent and verifiable way? This to me seems like the challenge. I am not as up on the literature here. Up to this point, there has been a pretty decent arms race. By definition, AI will continue to get better because — especially for visual models, for diffusion models — the way they work is that you have detection oracles that they test against. Certainly, you can never have something open source because it will immediately end up in the diffusion model. But for closed-source oracles where then you do not make it easy for them to test the oracle, one of the key things you do in trust and safety is then you have delayed enforcement mechanisms so they can’t brute force and test. I think it is quite possible. Let’s say I was still at Meta. My advice to them would be to also explore a variety of different detection mechanisms outside of the content itself. I would do the best I can on the content, but I would also be doing as low level access as possible to the camera. I would only let content come from the camera. And what I would say is, Nilay’s completely correct, there’s a massive hunger for, especially among young people, stuff that is authentic. I have three teenagers and they hate AI. They hate AI slop. They have a massive negative reaction. My wife and I created just a cute little family AI thing and they yell at us about a bunch of stuff, but they lost their minds about this cute little family AI thing because they just hate it. So I think somebody is going to come and disrupt Instagram and it’s going to be because… There’s already, I’ve seen these little things where it’s like you take the photo during the day, be real and stuff. There’s these little things, but somebody’s going to massively disrupt Instagram because they’re taking the images of the photos straight off the camera and then all the editing is going to be only within the app and they’re going to do things to make sure that nothing can make it into the pipeline from AI. You’re going to allow some kind of CapCut-like functionality that never has an AI component. I think that is going to be a killer app. Man, I could do another whole hour with you on this because the system you’re describing there is not detecting AI content, it is verifying real content. It’s much easier. Verifying real content is much easier. And there’s been a big focus on this on the professional side, like Adobe has a whole thing where if you’re a New York Times photographer, you have a $5,000 Nikon or Canon camera, you take the photo and it’s cryptographically verified. Yeah, but the iPhone that I’m holding here has that in it now. Right? Like there’s a lot of — Right, the iPhone has it now. Yeah. There’s a lot of this out there, but it’s the other side that I’m thinking about where it’s “We are going to detect AI text.” The leading model companies because of European regulations are now futzing with the text they generate. Text is much harder because you just have so much less data. Especially in 140 characters, what do you have in there to detect? The more you’ve got, the more you can do. It is amazing because I can detect Claude a mile away. So maybe I’ll just rely on my own human ability. Yeah, but not in tweet length. That’s true. The cheap ones, I can. But I’m just saying the notion that we’re going to verify what’s real instead of detecting what’s fake feels very important in the context of the media we consume. Detecting what’s fake is always going to be probabilistic and we’re going to asymptotically approach impossibility, whereas detecting what’s real can be cryptographically perfect. I would not pick the battle that we will eventually lose. I would pick the battle that we can cryptographically win. That would be my suggestion. And there’s somewhere in there where you have to build an entire chain of custody. We’re going to take the sensor data off the camera. We’re going to put it in a cloud server. We’re only going to allow edits on this server. That file will never get monkeyed with. Sure, that’s true for the New York Times and Adobe Photoshop and stuff, but not for Instagram. For Instagram, you can do it all because it’s only the iPhone and it’s only Instagram. So the entire chain of custody is within your app, it is within the iPhone sensor and it’s in Instagram. So it’s actually easy for them. Yeah. Look, like I said, We could do another whole hour on that because it feels like right next to the cybersecurity problem is the social engineering problem, and right next to the social engineering problem is the fake media problem. No, absolutely. How many phishing attacks are just “this website looked pretty real”? Oh, yeah. Or this person is talking to you or there’s an AI voice on the other side of the phone. And all of that comes down to whether you can detect what’s fake instead of verifying what’s real? And I don’t know the answer. Oh, no. I’ve had to train my parents that, “If you get a video from your grandchildren saying they’ve been kidnapped, you have to call me because the odds that it’s real are not very high.” That’s where we’re at as a society. I feel like I want to end on a hopeful note and I don’t know how to do it from there. Wrong guest for that, buddy. This is why you only come on every few years. Now after this, we’re going to do the series of birthday party guests. You’ve been talking about the dark ages. We’re in it, where it’s starting to accelerate. What are the markers people should look for in terms of progress to say, “Okay, we’re climbing our way back up”? AI is great for lots of things. I think AI is going to enable individuals. We’re going to see a huge explosion of new companies. Stanford students have always been entrepreneurial, but the number of my students who are now starting two-, three-, four-person companies who are doing incredible things, who used to need to hire 50 or 100 people and take a ton of venture capital, is incredible. We’re going to continue to see that. One of the things we hopefully see is a downtick in the number of major breaches at some point. We should hopefully see a downtick in the major CVEs that are wormable and that are in the fundamental infrastructure pieces kernel wheel. Like I said, I want Linux to be rewritten, but hopefully we see a downtick in the Linux kernel and those other kinds of libraries, openSSL, glibc, things like that. Maybe I should vibe code this up, but I think it would be cool to have an index of where we are in the cycle, of like an artificial index of all of these, the Dow for all of the libraries of where we’re going. Plus right now, Patch Tuesday used to be like 150, 200, got to 300, 600. I think the last one was 900 something, so we’re geometrically growing. It’d be nice to see Patch Tuesday start to go back down. Apple patches are the same thing. Yep, you inspired me. I’m going to create a Dow index for this, the dark age index. And then maybe we can publish it on The Verge. Yeah, that’d be great. But I think these are the kinds of things that it would be nice to see a reversal for. What would be especially nice to see is that the day or actually like 30 days before you get the next version of OpenAI and Anthropic cyber models, that you don’t have this huge round of patches because what’s happening is those guys are running their models against Microsoft’s code, against a bunch of open source, and they’re giving all the bugs away and you don’t know where the bugs are coming from. It’s just like anonymous reports for 1,000 bugs in the Linux kernel. And then a month later, here’s the new Astra Cyber. So if those get smaller and smaller every time, then it means that we’re starting to reach some kind of steady state. You should definitely do that tracker. We will definitely publish it. Alex, it has been incredible to have you on the show. We’ll have to talk again soon. Yeah, thank you so much. I think you do some of the best journalism on this stuff and I really appreciate it.