The Era of AI-Led Vulnerability Research With Tom Ptacek

But like the like the thing you said was like why you had me on to talk about this whole AI thing. A disclaimer I want to give upfront is I have like, I think I have kind of a lot to say about like the intersection between vulnerability research and AI, but like of all the people working on this, no one has done less than I have, right? Like I'm, I think the right way to think about it is that my interest in this is kind of journalistic. I have done stuff right? Like I am actively doing vulnerability research with LLM models just because it's fascinating to do, but I'm, I'm not like authoritative on any of, I mean any of this.

And I think that's going to matter when we get into what I'm talking about because I'm making a lot of, I don't know, spicy predictions. Yes, there are some very spicy predictions in your post, which is, you know, the kind of post that I love. But this kind of stuff like I feel like this whole space, the the usage of LLMS and AI tools to do like this kind of deep vulnerability research and exploit development. You know, people were talking about it, you know, maybe a year ago, 18 months ago.

But it at least to me, it feels like it's just accelerated really, really quickly in the last, I don't know it three to six months. Does that sound right? Does that. Well, I mean, when you say three to six months in AI years, that's approximately 75 years. Yeah, right. Like I'm, it's kind of like it's kind of a dumb dad joke to make, but it is weird when you leave security aside for just a second and look at software development where this has already been incredibly disruptive, right?

It's weird if you look at the timeline, like we've, we've known about LLMS, transformer models, like large scale Frontier models and all that. We've known about that for, you know, some number of years, but it was only like at the beginning of last year that Frontier models started to be really viable for serious production software development. Like a switch flipped in like autumn of the year before last. I forget what year it is because I'm very old, but I think that's 20/23/2020. 4 maybe, yeah. Yeah, 2020.

See, time is a flat circle, but like it's only been about a year during which all of the meaningful stuff that's happened with software development agents has kind of unfolded, right? So when you're, when we talk about like things being like when we talk about a shift being three to six months old, like you're talking about like almost half the amount of time it took for all of the software development stuff to roll out. So I think like a really important thing is just everything happens really fast. Another really important thing is just the switch that flipped that actually made software development with Frontier models workable.

I, I wrote a pretty like a, a post that a lot of people talked about in, in software development about how all my friends were AI skeptics at the time and like things were crazy and like, like software development with agents was clearly working. And at the time I had, I had claimed that like the big switch that had flipped to make that work was tooling was the fact that we like, we finally figured out how to build agents properly. And that like once you actually, once you actually solve the, once you, once you crack the code on agents, like the frontier models are much more, much more valuable.

But I think I was mostly wrong about that claim. Right When I said that it was like the big thing that shifted that's moving all of software development to AI. When I said that was agents, I don't think that's true. I think what really happened was that the models improved like the, the models got much better at tool calling and at doing multi turn conversations with tools in the mix. And just like doing like long term, like task solving that involves multiple steps and chains of different operations through lots of tool calling. Like that didn't work so well at the beginning of that year. And by the end of the year it was working well enough that we crossed a threshold and now the stuff was just really good for for software development.

I think the same thing is basically happening with vulnerability research. And I think like we're not prepared or how disruptive it's going to be. I think it's a much bigger deal to our field than it was to software development. And it's basically the biggest thing that's ever happened to software development, so. That sounds about right. I was going to. I'm glad you sort of explained like what that switch flipping was that you mentioned because I was going to, I was going to ask that, but I don't pay as close attention to that software development part of it as you do or as close as I follow the security stuff. But folks I know in that world kind of had the same line of thinking that you just delineated, which is, OK, this is cool.

I don't know how useful it is yet. Oh, it's kind of useful. Oh, shit, this is really useful. It's saving me a lot of time. It's actually writing good code. It's not turning out the shit that it was turning out like, you know, even two months ago, three months ago. And that very closely mirrors what I've heard from like security researcher friends of mine and yours that are just like, OK, I could it found a bug, but it's kind of something that I would have found on my own, you know, or the the bug isn't a real bug or it's not exploitable or it's in a weird place. And now in the last like two or three months, people are just like, holy shit, it found a whole pile of 0 days that, you know, I might not have ever found. And these are the smartest people in this business. Like the core story in software development is kind of like veteran software developers who are in it for the love of the game, who actually like writing code. Like we, we've had a critical

mass of those people converting from my day-to-day as writing code to I literally never look at code anymore. I just tell an LLM what to do. And maybe I like scan the PR really quickly. But you know, when I, when I push whatever the changes are that it made, I'll look before I hit the merge button, right? But like people who's like day-to-day was writing code and now never write codes simply do not write code anymore, right? How fast that conversion happens for those people. Like you go from I don't know about this AI stuff to literally I no longer write code in the span of a couple of weeks. That is a real story.

It happens over and over again. It's like, I think it's the big story in software development. I'm, I'm not like an AI maximalist, although through this whole conversation, I'm going to sound completely like an AI maximalist, but that's actually not where I'm coming from. I'm just seeing kind of what I'm seeing happening. And yeah, I mean, I think that that same shift is coming for vulnerability research, but I think it'll be more disruptive to vulnerability research than for software development. Yeah, it kind of seems that way, like just a very small, like the tiniest possible sample size. I when I was at RSA a couple weeks ago, I had a random conversation with this older couple like in the airport and they mentioned that their son was a computer science major who had graduated this PAT like about a year ago by the like. After he graduated, he kind of like didn't look for a job too hard.

By the time he tried to look for a job in the fall, no jobs. Like this is a guy with ACS degree who had had like a decent number of internships and all that. Companies had nothing for him because AI coding had already like essentially eliminated any jobs he was qualified for. You know, it's really, yeah, it's really hard to predict what this stuff is going to do for the market, for employment, like for the profession. So like, I think a lot of what I say sounds grim for the profession. And it might be right. Like it might be really rough for our field as like a professional thing that people do, or it might not be right. Like, it's also possible that, like when we find equilibrium with this stuff, really what's gonna happen is we can do so much more, so much faster, that like more problems open up to people doing software security work or more problems open up to people doing software developments. But like, the thing I'm pretty certain of is that it's gonna be really disruptive.

And disruptive is one of those marketing weasel words that people say a new thing is disruptive. I wanna be. This is one of my things. I've been in a lot of arguments over the last couple of days with other old school vulner researcher friends of mine who are like not thrilled with the, the article that I wrote. And it's like one of my big things is I just want to make like, I want to put stakes in the ground, make real predictions about like what could actually happen here. So like disruptive means like this. The, the group of people that are doing vulnerability research now, it's not going to be the same group of people.

It might be a much bigger group, although I kind of doubt that or it might be a significantly smaller group. Their day-to-day work is going to be very different. Like a simple, you know, a simple thing you can say is that like most of the vulnerability research stuff that happens, say a year from now, most of that work will depend on, you know, LLM agents doing the heavy lifting. It won't just be like assisted by it in the sense that I can go ask ChatGPT how like the block storage system in the Linux kernel works, right? I don't mean that. I mean like the actual hunting for vulnerabilities or the actual tooling for weaponizing those vulnerabilities.

That'll be done with agents. Yeah, One of the things that sort of struck me in the in your post, in something that I think has been true for a long time in the vulnerability research world is like the talent pool of people at the top end of that world is relatively small. Like they're, you know, you could, you know, maybe not right now, but say 15 years ago between the two of us, we probably could have named 90% of the world's great vulnerability researchers. You know well the ones we know about.

Yes, right. There are plenty that are just we don't we've never heard their names. But with the advent of these tools like that pool is expanding, you know. Yes. Astronomically everyday, like it's just, you know, the and those people may or may not have the background and skill set to go and find those bugs on their own the way that say, you did or Dave Itel did or Katie Moussouris did or whoever it happens to be. I like that you're lining me up against Dave Itel, and I think it's actually the right comparison to make. I think that like if you want to, if you want to see where I am in vulnerability research in my field, I think Dave Itel is the most lateral person to me, right, in terms of getting vulnerability research stuff done. David, that is my claim.

One to one comparison between you. And one absolutely 1:00 to 1:00. Yeah, people, if you don't know, go look up Dave's record, You'll you'll see it's pretty good. But I, I wonder, you know, if we have this conversation even six months from now, let alone a year from now, what kind of results we will have seen from, you know, AI enabled research or however we want to phrase that. You know, the claims that like Anthropic put out with this Claude Mythos release that they had this week where they said, listen, we, we have this model that we let a bunch of software companies or a limited number of software companies use for a while.

It found a pile of 0 days in all of their products. But we're not going to let you have it, you know, we're not going to let you play with it. These companies all know what those bugs are. We've disclosed them that sort of thing. But I mean that's all great and it sounds a little PR marketing hand WAVY to me. I don't know but. So I think, I think the mythos stuff is a big deal and we can get into the mythos stuff, although I know less about the mythos thing than a lot of other people that are talking about this. Like, I know what it is and I have like, you know, some some decent understanding of how big of a deal it is.

But like I would, you know, when people talk about mythos being the big thing, I would, I would kind of roll the clock back. Maybe I think it's about a month ago. I would say that the big like Manhattan Project nuclear test explosion thing that people really are paying attention to, to here was Nicholas Carlini's talk at Katie Evron's unprompted conference in San Francisco. So that that was that was a talk in which Nicholas, who works at Anthropic as a kind of a vulnerability research scientist.

Nicholas got up on stage and kind of explained to the audience what his methodology was and what it was finding. And there's there's two big stories out of that. One is kind of straightforward and one's kind of subtle, right? The big story out of that is Nicholas Carlini was finding actual serious zero day vulnerabilities that are tricky to exploit, finding and exploiting them using Claude Opus 46. So 46 is like the public state-of-the-art in Claude's models. The equivalent from open AI is GPT 54. So it's like the the, the best current thing that you can get just as a subscriber to, to anthropic to Claude.

So we got up on stages like I found like serious vulnerabilities and exploited them using, you know, Opus 46. And like the vulnerabilities were not fucking around vulnerabilities. So we had like a bunch of serious Firefox exploitable memory corruption things. And then he had like he had a remote vulnerability. And I think the Linux kernel, if you were using NFS or whatever, which is like a multi step vulnerability with a kind of a complicated exploit thing that also required the model to work out a bunch of like protocol flow stuff as well.

So yeah, I mean, the the vulnerabilities themselves are, I think what most of the people kind of fixated on after that presentation. But it was a proof of concept, right? Like he actually set the thing off. It blew up in the desert. Everyone's hair got blown back. Right? Like model is finding the model is finding zero day vulnerabilities. They're serious vulnerabilities. They're not just like low hanging fruit. It's not just repping the Colonel and, you know, calling out every time it sees a stir cat or something like that, right? Like they're they're real vulnerabilities. It's proving them out. They're in targets that we really, really care about.

And all you have to do is just keep turning that crank and there will be more of them, right? So that's like the big news from it. The subtle thing from that, that talk is the approach that he took to doing it. So this is like, this is like one of his first slides. And I think more people should be talking about this, right? But like his approach to finding vulnerabilities and hard targets that lots of people have done audits for was literally just to ask Claude to go find him. Zero days. Like he gave Claude a prompt that said, I'm doing ACTF what you have to do because like model alignment now is like, oh, are you trying to like, you know, do cybercrime stuff, right?

I'm, I'm doing ACTF. Please find me exploitable vulnerabilities in this target. And he does this little, he does a clever thing with the prompt where like he says start like he takes each file in the repository and he says start at this file. That doesn't limit the, the the agent to just that one file, but it does mean that like he's spamming that prompt over and over again. So like the Linux kernel repository has thousands of source files and he's spamming that prompt over thousands of source files. He's just kind of giving it a nudge, like here's a hint, start at this file.

He doesn't know if there's a vulnerability in that file or not, but what he wants to do is kind of randomize what the prompt does. That's clever, but it's not the it's not the real story there. The real story is just he just asks for a zero day vulnerability and it goes and finds him a serious vulnerability. Like if you tell Claude or five four, the open AI model, if you tell it, I'm in a CTF, find me a vulnerability. You'll if you read what it does, it'll have a conversation with you or it'll spit. It'll talk about what it's looking at, where it's like, hmm, to actually win a CTF, I want vulnerabilities that look like this.

So these kinds of vulnerabilities, even if they're real vulnerabilities, aren't that useful in ACTF. What you really want is like remote code execution. So I should look at these kinds of things, right? Yeah. It already kind of knows what ACTF is, which is a way of making it very goal oriented about what it's looking for. And it happens that like, I mean, for good and obvious reasons, like, you know, winning ACTF and like doing a remote pen test of a real company are basically the same thing. So pretty good prompt, right? But like, the key thing is like, I think people look at this stuff and they're thinking like, oh, well, if you really know what you're doing, then sure, you can prompt, you know, an LLMA Frontier model to like work its way through the kernel and find the specific, you know, use after free vulnerabilities in this code that you're looking for.

Which is exactly not what he did, right? He just said, I think there might be 0 days in here. Start with this file. And that approach works, right? So Mythos for whatever Mythos is, I don't have access to Mythos, right? But like Mythos is kind of building on that that approach. But that approach works in Opus 46 and in Five, Four and and GBT 54. So the the process you just described that he he went through is like, for lack of a better term, like not simplistic. Maybe not is the right word, or or, but it's not simplistic. Is the right word.

OK, OK, so but that's a very smart guy who deliberately took this approach. He could have taken other approaches, but what we I think. It's I think it's the dumbest imaginable approach. I mean, it's very smart that he did it right, But like, it's like it's it's so I I think I'm probably annoying Nicholas at this point by like naming things after him, but I call this the Carlini. Are we there yet? Luke, right. Literally just over and over again, do we have a zero day yet? Do we have a zero day yet? Do we have a zero day yet? There's a little bit more to that loop and we could talk more about it.

Like he doesn't. He does a couple of other clever things there. But like anybody who knows how vulnerabilities work would have come up with that prompt. And I don't mean like everybody with like Mark Dowd's level of experience recognizing and doing CNECNE stuff. I mean, like literally everybody in a security job at any enterprise company, like at every like regional insurance company in America knows enough to build that. That harness that tooling. OK. So that's the kind of output that we might see, you know, in months or years down or weeks or months down the line when a lot more people decide to start doing this kind of thing is the way that it seems to me. I think that's basically locked in. I think with what's already out there in public, it's already locked in.

I think a good thing to pay attention to in all these conversations is if you work for a frontier lab or if you're trying to do new research that's gonna, you know, showcase it Black Hat or whatever, you have an incentive to go after interesting high status targets, right? Like a Chrome full chain vulnerability is like the sine qua mount at this point, right? Lots of people are going to want to get that done and presumably the Frontier labs security teams are going to demonstrate that. Like that is the thing that you demonstrate. If you want to demonstrate that it's over, like the models can do everything.

It's just like, you know, if I can push a button and get Chrome full chain exploits, like what else is there to talk about, right? But the stuff that Carlene he was doing and other people as well were doing with Opus 46 and even before Opus 46, right? Like they're, they're, they're finding vulnerabilities in like the Linux kernel. And lots of people have an incentive to go find vulnerabilities in the Linux kernel. But there's like a world of other software that is much, much easier, you know, to find vulnerabilities in than the kernel where like a lot of those LPA, lot of those vulnerabilities are like, you know, local host LPE vulnerabilities that are useful in jailbreaks and things like that. There's a whole market for all of those vulnerabilities. Like there's no market for vulnerabilities in backup software or, you know, printers or whatever other random pieces of junk are connected to your network. Nobody is looking at them.

So like right now they're at like they're at a threshold of security that's kind of based on the level of elite attention that was looking at them before, right? So we're all kind of coasting off the fact that everyone knows that, you know, Mark Dowd's team isn't going and finding printer vulnerabilities, right? Like, you know, mid level consulting teams might take a run at them during a pen test or something like that, but there isn't like a whole lot of serious vulnerability research happening there. But like, you know, the cost of that serious research is now 0 right? Like IA model will replicate it for me. So so I think pay attention to the fact that people are going to keep calling out like really high end targets, like somebody's going to get an iOS vulnerability or some home vulnerability and things like that, right. But those are targets that are like that's like the Olympics, right?

But in terms of actually moving shit around and making shit happen, like there's a lot of soft targets out there that like right now I've just been like kind of coasting on the fact that no one is doing serious like memory corruption vulnerability research on them. And now that research is free. So yeah, I think it's a huge deal. And just in terms of operational security, like what to be prepared for. So here's the thing, right? So I, I said earlier, I, I work for a company called Fly dot IO and we're a hosting company. Our big thing is, you know, we let people kind of run containers inside of virtual machines that we run for them. So all of our customer workloads are like sandboxed into little micro virtual machines.

They run on a virtual machine hypervisor called Firecracker. Firecracker is the engine behind AWS Lambda and AWS Fargate. So it's an open source project that AWS built, it's built in Rust. It's memory safe, right. So it's a very safe. It's deliberately minimized. It's the stack that you would design if you were going to run a hosting company. And also you knew who Mark Dowd was. And over the last couple of weeks, we've looked at every single commit to the firecracker tree because every single one of those could be, you know, they won't be labeled. This is a horrible security vulnerability, right? Like Amazon will release a bulletin at some point, but like you got to watch the repository beforehand. But there was a vulnerability in firecracker, the first one we've seen, I think since we, I've been at fly since 2020. This is the first one we've actually seen. OK.

The first serious vulnerability in the firecracker hypervisor, which again is Rust and memory safe, and they found a memory corruption vulnerability in it that allows you to overwrite host memory from inside of a guest VM in this desk. We happen not to be using a firecracker option that it depends on, but still the idea that firecracker was like a thing that we need to be worried about. We're worried about everything right now because somebody out there like nobody's thought of aiming a frontier model at doing vulnerability research on all sorts of random components that are in our, our organization. Like, I don't know, there's probably like 0 dig Grafana vulnerabilities for all of our targeting and graphics stuff, right?

Like we're laughing and I think it is kind of funny. And also you would need more than a zero dig Grafana vulnerability to get our Grafana. But still, like all of those components are now like, who the fuck knows, right? Like more than that, we all know we're all kind of fucked. Like there's a whole bunch of vulnerabilities coming down the punch. Yeah, that was the honestly the the most interesting point that you made in that post was that sort of attention scarcity thing that a lot of software either consciously or unconsciously is depended on for not for nobody finding bugs in it, you know, guys showing up to pone to own or whatever every year and finding the iOS bugs and the Chrome and the IE and Edge and Firefox bugs. That's all great.

As you pointed out, those things auto update, they're patched within before you hear of the bug, it's patched. But the millions and millions of open source projects out there that the world depends on and you know, live as dependencies on in all kinds of other projects that have not had that kind of attention piled onto them are about to find out. And yeah, I mean, we're, we're, we're approaching a point where you could just automate that, right?

Or you could just, you could write a script that's just gonna pull every package down and then do the vulnerability research. Right Now, it's expensive to do that. Like, like there's a whole thing about like the token cost of finding these vulnerabilities. And obviously that's not a problem that researchers at Anthropic have, but it's an issue that other people would have you get into. Like, what is it? How material are the token costs? Like how much would you pay for, you know, an automatic, you know, remote code execution vulnerability and whatever your target is, Sure, it might be pretty high, but like there's a, there's a whole thing about like the token cost for these vulnerabilities, but those prices will come down. You know, whatever, whatever Mythos is doing right now, at some point in the not very distant future, like Chinese open weight models will do the same thing and the token cost of accomplishing those things will come rapidly down. And yeah, we'll at some point get to the point where you could just automatically go sweep the entire ecosystem of open source packages, not even have enough

intentionality to pick a package. Just aim at everything, like try to find whatever and then just triage the the vulnerabilities that come in. Like, do I care about this vulnerability? Where can I use this? But your, your new starting point is I have this basket of vulnerabilities and everything and like, how do I best use them? It's not like this weird little, you know, node JSNPM package happens to be used by 50 things. People find a vulnerability in it. It's here's a vulnerability in this weird NPM package. Let's see how many things depend on it. Yeah, I wonder, I have two, two questions, one on the defensive side and one on the offensive side. I'm wondering what, and I know that teams at these companies are thinking about this for sure, but how the large vendors like Microsoft and Apple and Cisco and whoever else are going to handle like they already get hundreds or thousands of bug reports, you know, a month or would get pick a time period. They those triage teams get overwhelmed with stuff, a lot of which was trashed before, you

know, AI research happened anyway. I wonder how they're going to be able to handle the influx of these kind of reports. I think there's an interesting conversation about what's gonna happen to bug bounty programs. Yeah, or whatever the programs are that like that the vendors are using to pull in all those vulnerability reports, right? Or the major vendors are, right. Like the economics and the incentives behind running a bug Bunny program are already kind of sharply changing like the the most competitive people and most bug bounty programs pretty soon, if not already, will be AI assisted, right? And then what are you, what are you rewarding in the bug bounty program?

Like if it's just AI looking for vulnerabilities, just have a team internally that's doing the exact same. Thing. Yeah, what's the point? Yeah, I don't know. I'm not like a, I'm not like a major bug bounty program person already. And bug bounties have also like weird incentives that people don't talk about. Like they're, it's not shady. It's just like there's engineering reasons why bug bounty programs are run the way they're run. So like the most obvious reasons to to run bug bounty programs might not be why they're running them to begin with, but I can imagine that being a weird thing there. I'm not like, I'm not that worried about about like Google and Apple and I guess Amazon, you know, companies like that, I'm not that worried about their ability to keep up with, for one thing, they're well staffed and they're places to work and people like working there. And for another thing, like in theory, if they're kind of if they're firing on all cylinders as these vulnerabilities are coming in, they're closing off not just bugs, but whole bug classes, right?

They're able to make architectural moves in their code to say like, oh, you found a memory corruption vulnerability in this component. Well the problem there is that that component is memory unsafe and is also like not sandboxed or whatever. And like they'll make changes that'll knockout like a zillion bugs all at the same time. I think the open source thing is so much more interesting. They also, like the the major vendors are if not already completely AI pilled, are on their way to. Being oh dude. So yes. Yeah, like the Mythos announcement is like a multi company kind of coalition of big vendors that are using, you know, clawed Mythos to get this stuff done right.

Like that works in in vendor land. Vendors will pay for the tokens no problem. The surrounding error for them, they have developers that can make big changes in their code. My optimistic look at this is that like, if this plays out the way that I think it's going to play out in terms of like the huge influx of vulnerabilities we'll get. What this will really do is kind of hasten a shift towards memory safe software where like the industry has basically already agreed that all this C and C++ stuff needs to get either, you know, deprecated and sandboxed or replaced with Rust and other, you know, memory safe programming languages, right? And like they've been kind of on the way to do that for a while, but it's been like there's been a lot of economic thinking and how they're doing it.

Like which components do like have the most bugs in them or how do we triage this work? What, what do we prioritize and all that. And like, my optimistic take is just that like that was sensible thinking before and now like, OK, they're just going to expedite a lot of that stuff and we're going to get much, much more memory safe software. I get the much bigger problem as an open source and like for a lot of reasons, obviously resourcing is one of those reasons, but another reason is that it's still not the case that open source people are, you know, hyped about Asian based AI software development. In fact, a lot of a lot of major projects and a lot of very, very talented open source developers are actively hostile to it. So like they're playing catch up against, you know, adversaries that are just going to churn out vulnerabilities. It's like one of the things I kind of like about this moment is as somebody who's been kind of in an industry argument about how big of a deal AI software development is going to be. It's hard to, it's hard to win an argument or even really have an argument about whether AI

software development, whether AI building code is a good thing or a bad thing. Because people will just look at the whatever code you show them, they'll say that's not good code. Here are like 10 reasons why that's not the best code that you could have written or whatever, right? Like a No true Scotsman thing. You'll never win the argument, right? There's no argument with the zero day vulnerable, but there's no argument with remote code execution either. Works, yes. Or it doesn't. Memory outcome like you don't want it was. There was after the mythos announcement on Hacker News where I spend way too much time, a couple of different people were talking about how they were slop vulnerabilities.

This is the term for all bad AI software that's just slop. And it's like, what does that even mean? What is a slop vulnerability? It either works or it doesn't. Like if somebody's owned up your machine, OK, they slop owned up your machine. Your machine's still owned. Your way, yes, yeah. It's a thing I like about the moment is that there's clarity to it, right? Like there's really no giving around. If it's producing working vulnerabilities, that's real, that's there's no argument to have there. So I don't know, maybe that'll crystallize things for open source projects, but there's a lot of kind of in the community, there's a lot of resistance to any kind of embrace of AI stuff.

It's funny, the kind of the in the community, the flag carrier for all this open source vulnerability stuff is bad and dumb. And Slop was, what's his name, Daniel Steinberg, the guy who runs Curl. Curl is like one of the most important HTTP clients, big open source project. People pay attention to it. So Stenberg has been running a bug binding program for a long time and it's been a you know, he's had a lot of slop reports before and after AI, right? But there was a moment, I think earlier this year or at the end of last year, he's like, this is this is getting nuts. I'm just getting this constant torrent. Of.

Slop reports, where like a Slop AI vulnerability report is actually really, really annoying because it's going to be written in such a way that it'll look like it could be a legitimate report. You actually have to read it to know that they're like, they don't know what they're talking about. Right. In a bug bounding program, usually you can kind of. You can quickly eyeball like 60% of the bullshit reports out right away because they just don't make any sense. Like step one, bring up the Chrome inspector. Step 2, start typing JavaScript into it. Right. Like you can see immediately that it's nonsense. AI vulnerability reports are not like that. They're really good mimics of what like a potentially real vulnerability would look like.

So I can imagine it's like just a miserable time sorting through all those reports. So he closed down the, the curl bug bounding program and for like the last three, six months or whatever, anytime anybody talked about vulnerability research with, with a frontier model, there'd be like this Greek chorus of people saying Daniel Steinberg closed the the curl bug bounding program because none of this stuff is working. And then like, I had no idea that any of this was coming. So I get to take credit for this, this prediction. But like, I think the day after I ran that post, Stenberg like came out, like publicly said, like before, I was getting this torrent of slot vulnerability reports. But I have to tell you guys, it's completely shifted. Yeah, I'm getting a torrent of AI vulnerability reports and they're all real.

Like I think project after project is basically saying the same thing. And why wouldn't they? It's like, it's like obvious that that's happening, right? Like obviously if you aim a frontier model at a major open source project, it will find real vulnerabilities like your. Your new problem now is just dealing with the volume of good reports and how do you keep up with that? And a vendor will use, a vendor will spend tokens to do that. They'll have a whole process in place to get people to use AI bug fixes, as well as having teams of people that can make architectural pages, open source teams.

I mean, where are they even getting the funding to do the token stuff? I think that's an interesting question. I think, you know, some kind of industry thing needs to happen to, you know, arm the maintainers of important projects with, you know, enough model access to get this stuff done. But you also have to convince those teams that like, they're going to have to like they're in a race with people that are using agents. They have to be moving that fast as well. Yeah, that's right. I mean, there's been all kinds of those efforts over the years to like fund open source security, you know, code reviews and all that kind of stuff. But this is a different, different level of that. Like, you know, there's a, you know, whatever phrase you want to use, there's a clear and present a danger to them. You know, it's coming.

I'm really interested by the thing that you said a little bit earlier about how there's a big resistance in the open source community to using AI, you know, code assist or you know, I wonder if that has, and I know absolutely nothing about this. I wonder if that has something to do with the thing you started off talking about, how there's people that actually enjoy writing software that's. 100%. OK, 'cause they're like people like me who like to write are completely resistant to using AI tools to help them write because we enjoy writing.

Most people hate writing. They fucking hate it. So they don't do it. And they they're happy to have, you know, Gemini or copilot write for them. But yeah, I mean it. I guess that makes sense. Yeah. I mean, I think that's a, that's a big component of that. I mean, honestly, if I'm being hard headed about it, probably a bigger component of the resistance. Is that like none of these people are dumb? They can kind of all read the writing on the wall. Like if one person can get 15 times as much work done, if they just automate all that work, then the head count at a lot of these places is way too high for what they need to get done, right?

It's like, you know, they're, they're looking at automation that's going to take a lot of their jobs. And I think that's destabilizing for people for, you know, for good reasons. I have there's, there's no profession in the world I have less sympathy for in terms of like Labor Relations that software developers, like we've had it absurdly good for a very long time, but also like the whole premise of software development is to improve productivity. Like people have been kidding themselves for 40 years about the fact that this whole field is about automating away people's jobs.

But either way, I think that's probably the bigger component. But yeah, the other component is that like people get into, a lot of people get into software development because they like coding. And like I said, like the thing that happens if you're a veteran coder and you see how well this stuff works and, you know, you flip over to the dark side is you stop writing code completely. Like your life is now prompts and discussions. You basically become a manager, just a manager of robots, right? A lot of people deliberately don't become managers because they like coding, right? I'm, I'm sitting in my office right now looking across that. There's a table right across from me, right? Like I'm a bad amateur.

I'm a terrible amateur woodworker. Like we have a, my, my wife is the real woodworker in the family, but like we have a shop down in the basement and tools and all that. And I get like, I get what's attractive about building a table, but like, if I need a table in my office, I'm going to buy a table. Like it needs to be a table. It doesn't need to be a fulfilling artistic process of me building a table using like, you know, Japanese Sushimano joinery with like no fasteners and things like that. But that's what a lot of it like when you, when you read the arguments from people in, in software development about like why they don't like AI and all that, it's like the code is all very mediocre, which is true.

It is kind of mediocre code. It's much more, it's more reliable than the code I would write myself. It covers more of the corner cases and all that. It's better tested, but it's not especially elegant. And like, that was one of the joys in that job was coming up with really elegant solutions to things. Just like one of the joys of being a Japanese woodworker is building an entire house without a single fastener in it, which is like, it's it's great, but it's not the job. It was never the job. And people have been kind of kidding themselves about it. So yeah, there's a there's a lot I've never been hated, including in the instance where like I helped leak that Dan Kaminsky vulnerability.

I have never been hated more for anything that I've written than a post I wrote about how all my AI skeptic friends were saying crazy stuff. Yeah. I, I got, I got shit for that. But you've never struck me as somebody who really cares about getting shit. I do care. People think I write these things just to stir shit up and I don't think that's true of me. I don't think that's true. I I think you'd say what you believe and you know, come what may. I've had a couple of vulnerability research and people say, well, you deliberately wrote the vulnerability research. You deliberately wrote that the field is cooked because you wanted to stir up attention.

It's like, I just wanted to make a clear prediction and the clear prediction is all of this work is gonna go to the AIS. I, I can't say enough how much vulnerability research is actually a, a problem that AI should be better at than software development. Like the thing that AI in the, in the mainstream is best at it's most successful commercial use case, you've got like 10s of billions, hundreds of billions of dollars of spend on this stuff. And it's most successful use case in the actual market and industry is building software, is writing new software and finding vulnerabilities is something that is actually more specialized and more suited to AI.

And than building software is like that problem is just uniquely, exquisitely fit to what you know, a frontier model does. So I think there's every reason to believe that if the all of that stuff is, I mean, I, I could be wrong and that would be great. At least I like there was a clear question and I was wrong about it. But my take is like all of this work belongs to the AI. So right now, as you mentioned earlier, Tom, like one of the main limitations is kind of the cost of doing this like for a normal person, like the token cost is pretty high, you know, for for these models, like assuming that comes down the way that all these things tend to.

Do you see any other real like barriers or like what are going to be the limitations that are maybe going to prevent this from, you know, disrupting things as much as you think it might? Well, I'm, I'm a maximalist in terms of how disruptive I think this will be. So I do think that token costs will come down. It's, it's funny, I've been building software with, you know, with cloud code and with codecs for like the last year. I'd do a fair bit of it. But I talked to friends who like they were talking about the stuff that they were building and they would go into like, you know, $100.00 of extra token costs trying to get a project done.

And the entire time I've been doing this stuff, I've never even hit a usage cap before. I mean, I must be doing less than all my friends were doing, but like you buy like a or your company obviously pays for like a Claude account, like a 5X Max account or whatever, right? And I just, I never once saw the, the usage limits on it for writing a lot of stuff just never got there. And like one day of doing kind of Linux kernel vulnerability research, just file after file. You could, I could literally watch the usage meter climb in real time. Like just watch there.

Like we'll go from like 14% to 15%. It look like go grab a, you know, a cup of coffee. I come back, I'm almost at the cap, right. I get a couple of files in and then and I've hit the token limit, right. So yeah, it's pretty expensive, but those costs will come down and people will figure out. I think I sort of alluded to this earlier, but this was my thing with Nicholas Carlini's talk. And the second thing that he had there about how dumb his approach was and how effective it was, is I got really interested in vulnerability research with Frontier models like a couple of months ago, because you can build tooling to make kind of a not especially good model better at finding vulnerabilities. You can do a lot of the things a vulnerability researcher would do.

You can write fuzzers or you can like write abstracted program exploration tools or pattern matchers or model checkers. There's all sorts of like fun software development, like security software development work you can do to build kind of vulnerability research tooling, which is kind of what like people do in vulnerability research anyways. It's it works and it's fun to do, right. But like the models don't do that at all. Like the stuff that that Nicholas was doing, it wasn't relying on any special tooling. It wasn't running fuzzers, it was just kind of hot off the model. It's just printing vulnerabilities directly off of Opus 46 and I guess now Mythos, Right?

But even if you're stuck with Quen or other open weight models, you're not literally at the state-of-the-art. Like you're not using the literal frontier. You could do tooling development that would take a less effective model and maybe make it competitive. And we're in the very early days of figuring out like there's Claude code for building software and people right now that are doing vulnerability research with Claude are using Claude code to do it. Like the CI commander, you type Claude into your terminal and it pops up and you start telling it what to do. You could write a security version of Claude code that was tuned for doing security stuff with open weight models or whatever.

And, you know, who knows how effective that's going to be, but it's probably going to be pretty effective. Yeah, it's, it's crazy times. I, I know that so many of the vendors that that we've, I mean, you, you said it like they're AI pilled in a lot of ways, like especially the big ones. There's a, there's a lot of pushes inside these vendors to like use AI for almost everything. Like, not just like suggestions, like mandates, like you will use AI tools in your job or else, you know, and like there's going to be, there's got to be some sort of like logical end point to that where people are like, OK, but like you still need humans to do some of this work. Like, and I, I feel like that's true for vulnerability research too, to a certain extent.

I mean, like you can't just take the simplistic approach to everything. Like at some point all the shallow bugs will be gone, right? And you you need like marked out in Dave Itel to go find the the crazy ones. I think it's, I think that is a really big and interesting question, which is my way of saying I don't think that's necessarily true. I'm not confident enough to make a, a prediction about that. So I, I before I wrote the post, I talked to a couple of different vulnerability researchers, researchers.

And one of them had basically that point, right? Which is like models are really good right now at finding and exploiting vulnerabilities that are based on kind of the published science of vulnerabilities, right? Like, yeah, inside of Firefox, which Firefox is also kind of a low hanging target all by itself compared to the other browsers, Firefox is just easier to work with, right? But like there are kind of well known published techniques for like, if you have like a wild right vulnerability inside of Firefox, there's like a, there's a literature of how to take that right and turn it into a controlled right.

And then from there, how to like, you know, get code execution right. It's all documented. It's not documented in convenient places, but there are places where model will find it and build it into their, their training set, right. But there's a lot of unpublished knowledge about how to take low level exploitation primitives. So like I get a random wild right inside of Firefox and I want a controlled right. There are like techniques that are not public that people use to actually weaponize and make reliable exploits out of these vulnerabilities that the models don't know. And that was one big point that they had is that there's a, there's a, there's a non public literature of how to take, you know, kind of dumb vulnerabilities and make them workable.

And like, there's a big open question about whether LMS, whether frontier models will rediscover some of that stuff or discover new versions of some of that stuff or whether it won't. There's, you know, it's, it's entirely possible that there will still be a very high end of vulnerability researchers who find new vulnerability classes or new exploitation techniques, who literally set the state-of-the-art, like the state-of-the-art for exploitation will still be human. But that's a, that's AI. I, I really want to emphasize how small a section of vulnerability research those people are, right?

Most vulnerability research is dumb. And more importantly than that, most of the most impactful vulnerabilities are dumb vulnerabilities. Everybody remembers Heartbleed, right, Which was every web server on the Internet was like leaking all their secrets and all that. That was the dumbest imaginable vulnerability. It was really, really, you would have found that vulnerability quickly in 1995 if you looked for it, right? And the reason it didn't get found wasn't that the vulnerability was complicated. It was a dumb vulnerability. It's that it was buried in a mountain of like TLS code of, you know, you know, HTTPS and SSL protocol code inside of open SSL.

You had to know a lot about how open SSL worked and how the TLS protocol worked to find your way to that code and kind of trace the input to that code and see that like this mem copy right here was taking untrusted data. And like there actually was a really easy way to send that data to a web server and things like that, right? Like you had to know a lot, a lot of random shit to know that that vulnerability was exploitable. The thing about Frontier models is that they know all that stuff, all of it. Just latently it's built into the model, right?

Like a map of how open SL code works is already in all of the Frontier models, right? You just have to ask for it, right? So like the, the, there's a, there's a lot of like very impactful vulnerabilities that have not been published yet. Like you keep seeing people talk about these, these findings. Like I found a new open BSD vulnerability that had been in the code base for 27 years, right? Like that, that's a thing. You hear that all the time. Like this, this was in the code for years, like 5 years. Yeah, whatever. Right. It's because like the actual vulnerability itself is not especially interesting in terms of cleverness and difficulty to find.

It's like picking your way through all of that code and trying to keep all of that state in your head, trying to follow all the details that you need to follow to like clear a path to that that that code and find it and then know how to exploit it. And again, the models know this stuff latently. It's already built into like their understanding of the world. They don't have to do a whole lot of work to figure that stuff out. And if they do, if they do need to read more code, they're also very good at reading lots and lots of code. Yeah, and as you pointed out, they don't get bored, they don't get tired, they don't have kids to go talk to or put to bed.

They just like go and do. Sarah Connor said about the Terminator. We need to out to see them coming and all that. Yes, exactly right. You mentioned TLS, which made me, which reminded me to bring up the, you know, Tai Dong has been like in his new entity Caliph. I don't exactly know what they do, but I follow him on Twitter. So I see all these things that he's he's posting where they're find, they're using these tools to just find all kinds of crazy stuff stuff. He posted something the other day that like an intern that they have found some crazy RC vulnerability on like his first day using Claude, I think. And so for people that don't know Ty and Juliano Rizzo way back when, I'm not, I don't even want to say how long ago it was. It'll date us both, but came out with a couple of like insane TLS attacks, BEAST and the other one was crime. OK, this is kind of a crazy question. I don't know that you can answer, but like would those guys have like would those

attacks that they developed I guess is the right word? Would AI assisted research have been able to find the stuff that they found back then? I mean. Eastern crime, super interesting question. The answer is I don't know, but super interesting question. So those are kind of cryptographic protocol vulnerabilities. So that's like, that's not picking your way through the code so much as like looking at the transactions in a protocol and then knowing what the security boundaries of, of a particular cryptographic instructions are, which we're getting into the kinds of vulnerabilities that I actually do work on and like a lot, right. So I, I will say that Frontier models are very good at understanding like the detail, like a frontier model will do an excellent job of critiquing A cryptographic protocol. If you put a, you know, if you try to design A secure transport or if you're trying to build a new secret storage system and you have like multiple levels of keys or what block cipher constructions or how you're authenticating data, like they're very good at critiquing those things. So you know, whether like there.

So the, the thing, the thing about this is there are like frontier metals are also very good at cooking. Like if you want to have a conversation about planning a menu with a frontier model, they do astonishingly good jobs of planning menus. And the reason for that, it's obvious there's so much content for them to train on. Yeah, everybody has written like, for every possible thing you could cook, 50 different people have tried to get at the top of the Google search page with a different recipe for it, Right. They're extraordinarily good at cooking, right. So there are a lot of memory corruption vulnerabilities out there. And like, there's a whole cottage industry of like, claiming CV ES for them and writing them up and CTS built on them. And cryptographic vulnerabilities have always been this kind of weird corner of vulnerability research where very few vulnerability researchers do cryptographic vulnerability research.

And in fairness also, those vulnerabilities are super fun and super interesting. But even with BEAST and crime, right, BEAST is an incredibly cool vulnerability and it was impactful. Like everybody had to update their web servers. It was a big deal. But in reality, every serious memory corruption vulnerability of that year was more important than BEAST in terms of actually compromising people's security, in terms of being a thing you had to react to. So there's a, there's, there are a lot fewer cryptographic vulnerabilities to pattern off of. There's, there's a lot less of them in the training data. There's there's less to look at and you still see kind of like novel state-of-the-art vulnerability, like cryptographic vulnerability research and cryptosystems coming out of academia.

It's all, it's all still kind of new, which is one of the things I really like about it. My guess is that like you could, you could use an agent to help work out something like beaster crime, but you would be doing a lot more prompting. You'd be telling it a lot more about like, here's the protocol. Like look at this part of the protocol is this, you know, are the ways we're setting up initialization vectors here. Does that look right to you? Like if you said that to a model today, it would say, no, this is not good. The way you're setting up these chaining Ivs in CBC mode, you know, this is going to cause problems. Then you'd say, what kind of problems will it cause?

And then it, like it'll workshop with you all the different things you could possibly do to, you know, you know, to exploit that condition and all that. You would work your way. I think back to I should, somebody should try it. I should try it. I'm just being late. But like, you'd probably get there, but I don't think you could point it at TLS and say, find me 0 day in TLS and then have it come back with BEAST. I, I, I would guess that there's not enough training data for it to be good at that off the off the model. But again, I don't have access to Mythos, so maybe Mythos is really good at that. I hope that's true. It's entirely possible. Who knows? I mean, it's a black box at this point, but we may find out in the next few months.

I don't know. It's. It's a fun time to be in the field, the last moments of the field, true, like. Also, it would be a little bit fun to be at the end of the world too. It's kind of the same sense. I just, we may not be far off, Tom. I mean, things are not going well. Things are going really badly outside. That's why I'm in my viewpoint. Yeah, yeah. All right. Tom, thanks so much for your time, man. It was great to talk to you again as always and thanks for your insights. And for people that want to see more of your work, where can they find you?

Said fly dot IO is where you're working, but you don't publish very much stuff publicly these days. Hacker News. If you want, go read Tom's comments. No, there's, there's a right answer to this question, which is that if you are not tired of listening to me blather already, what you want to do is listen to Security Cryptography and whatever, which is podcast that me and Deirdre and David do about vulnerability research and especially crypto vulnerabilities. So if if if cryptographic vulnerabilities are your thing, I would recommend the Security Cryptography and Whatever podcast Deirdre's podcast that she lets me tag along with.

I would recommend that too. Good plug. All right, Tom, thanks so much man. It was great to see you it. Was good to talk. Thanks. Take care while. You were having the planet. I was trying to decide to grow up in land, even if I couldn't understand it.