What the Data Tells Us About Claude Mythos and Bug Exploitability | Jay Jacobs and Michael Roytman

data interfacing with a lot of large organizations. Kind of security was acquired by Cisco in 2021 and I was exposed to a whole bunch of those. And every single time we talked to a sufficiently large or complex or mature organization, the first pushback would be like, this model is great in the non targeted attack sense, but I am a special stuff. Like I have special data, I have a special environment, my threats are different. This doesn't capture my reality. And you know, with that hat on, my response was this is the best we've got. This is the model that is data-driven and others really aren't at the time.

I think in the back of my head, and I started talking to Jay about this probably two years ago, was what if we did build individual models per customer? What if we did incorporate that data about that customer and build a model specific to them? What does that world look like? And Empirical has really been us trying to realize that journey along the way. We got Ed Bellis to take a look at it and we're going to have him invest in the company. And he was like, no, that's, that's a bit too good of an idea. Why don't I join you guys? As if he didn't have enough to do. Yeah.

I mean, it's, it's hard to remember now, at least for me, but there was a time where kind of this data-driven approach to security like almost didn't exist. Like it was just, you know, and it wasn't, I mean, it's long ago in like security community terms, but it's not that long ago in absolute time terms. You know, like even just 15 years ago, like this was seen as like this sort of like out there idea of like, what are you going to do with data?

Who knows? Like how are you going to apply data to like bad guys, you know, trying to get into my network? What is that going to tell you? You know, we, we still run into that. Yeah. Oh, I'm sure. Mortality is absolutely still out there. Yeah, yeah. I believe it like do, but do you run into that at like what level of the organization is, is, where does that resistance? I, I, I don't think it's a level. I think it's just something about some people have a very skeptical bones in their body about data-driven security stuff. And so I've seen it at every single level and I've been surprised some people at every single level are like, this is the greatest thing ever and really want to get into the math and stuff.

And so it's, it depends. I don't think it's a specific role or anything. It's just a mentality for some people. Yeah, I think that's right because, you know, especially people I think of maybe our generation J, like people got into security 'cause they like tech stuff, you know, they like breaking things and they liked that sort of like, you know, back and forth type thing. Not because they like graphs and math, you know, certainly not what they were in for. And you would still see that, see that sometimes it like at conferences, you know, is it something like Black Hat that you mentioned, Michael?

It, you know, it was all the guys like trying to break stuff. And then there'd be like a group of like 12 of you guys over in the corner, like, you know, in like the data data world talking about like actual results and like, look, we can prove that like this stuff works, you know? Yeah. I think both are really necessary parts of the security ecosystem, but security's been growing. I think that part's missing from a lot of discussions about this that 15 years ago security, some parts of security were a human scale, let's investigate type of problem.

I think for about 5-10 years, it hasn't been that the volume has been such that you need to do statistical analysis and try to be more efficient with what you're working on. But once you figure out what's efficient, it's still a pretty manual effort of investigation and figuring out what to do and actually doing the remediations or mitigations. I like to think about just how firefighters rarely take a step back and think about like the statistical incidents of fires across the city because they're busy responding to the day-to-day, and I think security has been in that mindset for some time. Taking that step back is actually quite a luxury. Yeah, that's a, that's a good way of putting it. Yeah, I like that.

And it's so you mentioned the EPSS model, which I think is one of the coolest things that's kind of developed over the the last few years, which is the exploit prediction scoring system. OK, Yeah. So essentially like I'll let you guys describe like how it all works, but it's the idea is to like it what it says on the label, like tell you how likely this bug is to be exploited taking into account a whole bunch of factors and parameters.

But Jay, where did that? And I know it's like from the first group, the incidents incident response group. And yeah, so I want to go back to the beginning and, and Michael sort of glossed over a little bit. But when we started, when I, I started Santi with Wade Baker and we were contracting with Kenna and a bunch of other companies and Kenna had just wonderful data about the vulnerabilities in existence, the remediation, a bunch of extra data about vulnerabilities that you just don't find typically. And then we had another customer, we're working with Fortinet and we're producing Fortinet's threat landscape report.

And of course, Fortinet has, you know, I think over 100,000 devices deployed globally and they're all reporting the, the activity that they're seeing centrally. And so Fortinet had this wonderful amount of information about all this activity going on, all these CV ES being attempted to be exploited, blocked, things like that. And I was looking at the exploit activity and looking at Kenneth's data and kind of had some exploit activity, but we're looking at Fortinet. Like if I bring these together, we could we could model this like we've got outcome data.

The things that we want to stop is this exploit and then all the stuff about the vulnerabilities from Canada. And so bringing those two together was basically the birth of VPSS. So OK. That's no small thing, because if you really think about it, security is two types of data. In general, it's, I looked at a system and here's the state of that system, which is what a vulnerability is. It's like this exists in maybe production, maybe not production on a particular device or machine or database. And then there's the telemetry data, which is in real time. These things are happening.

Historically, they've been to sometimes even different groups in an organization. And I think what Jay did is essentially cross that divide of data engineering more than anything to ask the question, well, what what if we looked at all security data, what could we answer about an organization's posture in real time? So if you're looking at a given vulnerability, how does the EPSS like? What is it taking into account to come up with a a exploitability prediction score for that bug it?

Takes in as much as I can everything and anything we can get our hands on. But for the most part, I think a lot of the strong signals right now we've got about 2800 different variables feeding into the model. And I think probably 900 to 1000 are actually feeding and helping the the scores and stuff like that. So the extra is just sitting there. But anyway, a lot of it is like one of the key things we'll put in the vendor of the product and that vendor holds so much information. Like how prevalent is that vendor? You know, if you think of, you see a vulnerability and you're like, and then you see how it's a Microsoft on, you're like, oh, wait a minute, what was that vulnerability? You know, like you get these little tip offs and the same thing with the model.

And so we also do a ton of scraping for exploit code being published. Those are usually pretty good indicators. Yeah. I mean, just everything and anything different chatter. We try to pull texts of people talking about these vulnerabilities and then mine that text for keywords and phrases and all sorts of things that people are talking about. We try to get at the characteristics of the vulnerability. So things like CVSSI guess, but we try to get a little bit more granular than that.

So we try to get more detail about, you know, if you know, like as a sequel injection and what kind and all the details that we can get. And it's all just fed into this, you know, meat grinder of a model that learns all the patterns. And so when EPSS is rating something high, it actually EPSS, when we're scoring stuff daily, it doesn't know about recent exploit activity. So it's not actually factored into the model. And so when the model scores something high, it says, hey, looking at all the attributes to this vulnerability, it looks like it shares a lot of the attributes with things that we've seen exploited. And so it's saying like this has the the markings of things that should be exploited.

And that's what the model is producing that that's sort of, it looks like exploited stuff. You didn't use the key operative word, which is prediction, right? I think that's the difference between EPS and every other scoring system that came before it. It's that it is not describing the current state of the vulnerability, although of course that data is factored in. It is making a prediction about is it likely to be exploited or not. Yeah. And so if if I'm an organization, let's say I'm, you know, an insurance company and I'm looking at this, you know, to try and grade vulnerabilities, is it taking into account or do I have to sort of make these calculations myself? Like the industry that I'm in, you know, like how likely it is that, you know, insurance companies will be attacked with this type of bug, that kind of thing is, is that kind of context there?

It's, it's not there that that kind of breakdown because all we do like when we publish EPSS, it's literally just ACV and a score like that is that is the richness of our data there. And we've been doing that since we've been publishing since 2021. And so everyday just CV and a score. So we don't break anything out by industry or anything like that. We try to keep it global, just, you know, that initial sort of global indicator. Yeah, and and the the output is an actual probability and we actually calibrate that probability. So when we say, you know, like 2% seems like a very low probability, but it's a 2% chance that it will be exploited to one out of 50 vulnerabilities. At 2% we expect to see exploitation activity, you know, and so that's that's a key thing. If we talk about thresholds, I'm going to bring back probability into that conversation.

But yeah, we output a probability. OK, and I, I think I mentioned that earlier this, this Linux bug that came out this week that, you know, everybody's kind of freaking out about it's a local privilege escalation bug, which, you know, not not terrible, but it's got a name and so people freaked out about it. It I happen to notice today that the EPSS score for it was like .1%. Like I don't maybe you guys can explain that to me. I don't totally know like why that would be the case, but is it just because it's a local?

I'm not sure. That could be part of it and the fact that it's on Linux could also play into it. I mean, there's, there's several different things that can play into it. Maybe there's no, I don't know if there is or not, but XY code being published or haven't seen anything like that. So, so all of those have an earmark of something that is not likely to be exploited. Not saying it won't be, but it's just has a very low probability of being exploited. And the thing that we have to keep up on though, is to keep watching this.

So something like this, if it's garnering a lot, it's giving a lot of headlines. That means that exploit code probably isn't far behind. We're probably going to see some really detailed write ups of how to exploit it, you know, some things like that. And that's all going to start to shift that probability as we see that activity. And so that's why we're doing daily scores because that's going to shift. It's going to change as we go through time. OK, so this is this is a really interesting point that's super easy to explain in layman's terms for me, because I do this myself every day. If there's a weather forecast for a day that I really want to go to the park, I check it like 3 days before and that probability could be like 20% chance of rain.

But then the next thing I do is I check it the day afterwards to see if that probability has changed based on new information that's rolling in about the environment. And then right before I go outside to the park and think that like for the next three hours I'll be outside, I again check the weather forecast, which is the same exact technology as EPSS. It's essentially saying given the information we have historically about this system, dynamic system, weather system or vulnerability ecosystem, what is the probability that in the next 30 days is what EPS measures? The weather forecast is usually measuring something in the next couple hours.

Is it going to rain or not? Now that is a that's not a static thing and it's not a static thing because it's not a static thing in the world is weaponized code comes out as people embed this and something that is published and somebody else can pick up as they pair it with another vulnerability that can take advantage of the local escalation, that probability will change. So one of the downsides of the model is that it's making a forecast probability guess. One of the upsides is that you just can keep running it time and time again. Like internally we run them hourly instead of daily to see how that probability is changing. Do you guys ever run them against, like, historical vulnerabilities, you know, like Microsoft stuff that came out like 10 years ago, like that is is sitting there? Yeah.

We're scoring every single vulnerability that's been published into the CVE data. Data OK. And so yeah, we're scoring 1999 vulnerabilities still and there are things from I think early 2000s, maybe there's a 99 one too that is scored pretty high and we're still seeing some activity on really old stuff so. Yeah. That part is amazing to me. I think the peak actually is like 10 years, 9 to 11 years, mainly driven by Microsoft.

But like the peak activity are things that have been published 9 to 11 years ago that we're seeing for the activity, which is crazy. It is. I mean, there was, I don't know if it was last week or the week before, but SIS added something to the Kev that was from like 2009. It was, it was, yeah, some kind of Windows Excel bug, I think. And it's a good. Segue into some of the work we're doing for the DBIR that's going to get published any minute now.

So we analyzed re exploitation probability. Given that something has had exploitation in the past or was on a catalyst, what's the chance that it's going to get exploited again in the next 30 days? It's actually a really interesting question because some things on this, it's a catalyst to your point are 10 years old and some got on there three years ago, but we don't know what's happened in the past three years of that vulnerability. Yeah, I'll, I'll leave it to Jay to explain re exploitation, but that that result is pretty interesting. Yeah, it's, it gets really complex really quick.

But like the question I had is if you see something on like Scissor Kev's list and all you know is that it's on the list, maybe it was put on yesterday or three years ago, but all you know is it's on there. You don't know when it was last exploited. What is the probability that it will be exploited in the next 30 days or something? And I think when I looked and I was using our data and I did it over like the last six months, but it was something like roughly about a 60% chance of exploitation activity for any of them on the Kev. And then when I started looking at it very closely though, and I looked at our data and I said when was the last exploitation activity that we saw?

It was this beautiful little curve that just decayed from like 100%. So it was exploited yesterday. You got like a 99.8% chance it'll be exploited today. And then if it didn't get exploited then you've, you know, then it's like got a 97% chance and then like a 94% chance and, and it just drops down and sort of flattens out. And after about a year it, it's about the same. If we haven't seen activity in a year, you've got about the same chance of never been exploited. And so just knowing that recent exploitation activity was a really key bit of information.

If you know something has been exploited, it's really helpful to know when was the last time it was observed with activity being exploited. Yeah, that makes a lot of sense to me because there are, you know, 10s of thousands of known vulnerabilities in, you know, I don't know how many are in the Sisi cab, but there's enough. And, you know, attackers don't need brand new vulnerabilities all the time. You know, there's plenty of RC bugs in every firewall sitting on the Internet and all that kind of stuff.

And the interesting thing to me, and I mean, I don't, there's no real way to sort of quantify this, but you'll see things get added to Sis's Cav or other Cavs where, you know, say it was an iOS bug or a Chrome bug that was used in like one of those targeted attacks against a journalist in, you know, Morocco. It was chained with like 9 other nine other bugs and they add it which is cool. But like, the chances of that ever being exploited again are probably pretty low because it took somebody eight months to build that exploit in the 1st place and Apple just patched it and you know, that kind of thing.

So there, there's all that, that sort of like variance in there too. But yeah. Yeah. I mean, yeah. And we had. To admit to ourselves at some point that the technology of the models we use for non targeted attacks, like the Oracle Weblogic 2019 CV that's constantly being exploited is not the same model and technology that you would use to deal with targeted attacks. Right. And I think that that was not obvious in 20/15/2019. But to take it back to the beginning of those organizations that are like, I'm a special snowflake, my threat vector is different.

Some of them are right. They are special and they have targeted attacks that might get triggered against them. That's not going to come up in the data sets that we used to triage the generalized vulnerabilities that are used against everybody. Yeah. And I mean, nor should it really. I mean, there's kind of outliers or I mean, they're super cool to look at those exploit chains and all that kind of stuff. And like, think about the amount of effort that went into that and that sort of thing and how they're, you know, sort of driving innovation on the defensive side too, how Apple and Google and Microsoft and everybody else are working to break those exploit classes or bug classes.

But yeah, I mean, one crazy snowflake iOS bug shouldn't, shouldn't, you know, drive a bunch of, I mean, they drive headlines. That's, you know, we know that. So the the initial kind of idea behind this this podcast for the three of us was you guys reached out about the EPSS being mentioned in anthropics documentation.

I guess documentation is not the right word, but blog post about the Mythos release and I kind of missed it on first glance because right above that was a piece of advice that said, I believe patch everything that's in the Kev, right, Right. Which is like, have you tried not having vulnerabilities essentially? Like, yeah, your doctor telling you, have you tried not being sick? Sure, yeah, that's excellent advice.

But right below that was essentially pick a threshold, like determine your threshold for tolerance in EPSS and patch everything above that. Like, you know, so how did that come? Were you guys aware that that was going to be in there? Or was it just sort of like, oh shit, look at this? Yeah, and it's amazing that they use the language of threshold because one, it's the right way to think about any model, but 2, they would know they build models, but two, that question of what is the right threshold for me has come up on the EPS CIG no less than 100 times over the past five years. Oh, I'm sure.

Yeah, and I, I keep trying to punt it because of course I have no idea. You know, it's all about your risk tolerance, the security controls everything. You know, like there's so many factors that go into that threshold setting for for tolerance and, and early on people were like, hey, you're producing the score. Where's critical, high, medium and low. We want to, we want to know where critical is and, and high. And, and I kept on saying it's not like we're producing the score, figure it out like, you know, and it's just the other thing, like if you think about risk, you know, the, the pressure, the, the exploitation activity coming in and then your ability to resist it and then the outcome on the other side.

We're only measuring one of those three things, you know, it is not the whole risk picture. And so for us to put a threshold, like we're very concerned. I'm very concerned that if we say here's critical, then people are like, great, we're going to do this threshold and above, then we're done. I'm like, that's not how it works, you know, because I said like 2% seems low, but you're still going to see some exploitation activity on the low end, you know, so I can't tell people don't, you know, don't patch under this or something. So I've always avoided that.

I want to grandstand about criticals and highs in general a little bit. So this is a philosophy. We have an empirical where we don't want to tell people what higher critical is. We want to give them the tools to determine that for themselves. But what Jay saying is absolutely correct. There's for the typical large organization that's using 10 different security ventures. They have 10 different definitions of what high and critical is, some of which are correct and maybe that thing is critical, some of which are not applicable in that particular instance, some of which have mitigating controls already in place.

So it was a critical a priori. But isn't that organization? And so I have met many a person who spends their day-to-day explaining to a regulator or an IT person why something that is labeled as read high or critical is not read high or critical for them. And vice versa, something that is read labeled low is actually super important for that organization. Adding another layer of criticality in the exploit prediction scoring system. We'll kind of do away with the entire concept of probability.

We want to give people the tools to measure likelihood. That's EPSS. You know, a little bit of an empirical plug. We've got models that do take into account mitigating controls and we can build specific ones for customers whether more of the risk is measured. But even in those instances, it would be disingenuous for us to say we understand your organization's risk tolerance and posture and capacity. And so we're going to tell you to patch everything above this. Now we want you to select and have a discussion at maybe even the board level about what is our risk tolerance? How much risk do we want to remove from the organization? How much capacity do we have to do that? Can we really patch 5% of our vulnerabilities in any given month?

That is a, that is a measurable thing, right? Some organizations can that have insane budgets, others can't. Also 5% is going to change dramatically over the next two years, but that means the raw numbers. So just taking a step back from the the entire security industry has been kind of saying, I found 100,000 things and 30% of them are high. That determination is actually something that enterprises and users have to be making rather than vendors. But I I completely understand the impetus for it, right? Like, you do need to filter the signal from the noise in some way.

It's just that maybe the filter's in the wrong place. Yeah, that's a great point. I, I do think, you know, going back to your point about a lot of organizations thinking they're special snowflakes, like in some way every organization kind of is like they, you know, they have their own risk tolerance, they have their own budget constraints, they have their own, you know, resource constraints. And in terms of personnel and like time to get this stuff done even without, you know, frontier models dumping a giant pile of bugs on them in the near future. Just, you know, Patch Tuesday, Oracle, you know, Chrome, all that kind of stuff, not to mention whatever other, you know, software they're using.

Even well resourced security teams have a tough time keeping up with that stuff, you know, and so they do use, you know, we got to get everything that's critical or high slash important that that's what we're doing this month. Like we got to get those done by May 15th because that's what you know, is in my KPIs. Like, you know, like there's real world shit that determines that stuff. Right. Oh yeah, the SLA question is an entirely different one too. If I say I fix all criticals within seven days, that seven day period is also a risk and capacity statement that I'm making about those criticals.

But the definition of the critical doesn't include that seven day period. So you have two knobs to turn right? One is how quickly do I do things in this bucket? The second is what what is critical? What is that bucket? How small is it? Because if it's really just the 50 CVS that you know could really 'cause serious exploitation and are being exploited elsewhere in your environment, maybe you can't get those done in three days. But if it's 5000, then you can't.

And that three day SLA is actually just you nicely lying to yourself and others. Yeah, well, we all have those little lies we tell ourselves. Yeah. Some are about bugs, some are about the. Yeah, our sleep schedules and our diets. But yeah. So what are you guys anticipating with, like Mythos and these other Frontier models? I mean, there's been a lot of like there was a media backlash, you know, Mythos specifically, then there was kind of backlash to that of like, no, this is really cool.

There's a lot of like interesting stuff coming out of this. You know, I, I saw a blog post from the folks at Mozilla talking about like the, you know, I think it was 200 and something bugs that it was found in, in Firefox. Maybe I'm overstating it, but it was quite a lot. And how, you know, kind of game changing that was for them. I don't really know where this is going, but it seems like if we have this conversation, like even three months from now, we might be looking at a completely different landscape, you know? Yeah.

And for us like I, I've thought a lot about it and actually it doesn't matter what I think for for the modeling because the modeling is is self learning. You know, it like EPSS right now, I don't think I've looked at the data in like a month. You know, the underlying data, it is going out and grabbing and collecting all the stuff and producing the scores and it's reacting to what's going on and I'm not there doing anything with it. And so I think what the only thing that I can say that absolutely will happen is I think we're dealing with it going to be dealing with a different system. You know, like our system right now is, was the one that we've been working with for a long time is that the attackers have finite resources.

The, you know, their attention is by far the most valuable resource. And that that balance is going to be shifting and it already is shifting. And so the, the rate of exploit, the things that are targeted, I think all of that is going to be shifting. And I think the, the probably the hardest thing is for people who've been in the industry for like 20 years or something going to be like, no, I know what how vulnerabilities are picked. This is a bad one. And that's a good one. You know, like that's going to be really hard for them to shift their perspective, but for the model, it's going to be like, oh, I'm seeing this new activity or we're seeing it over here now we're seeing it over here, You know, so we're going to have to retrain more often. We're going to have to keep up on the model, but largely that model is going to react.

So, so the retraining part is where I was going with this. When we trained DPSS Version 4, which we released in March of last year, it had been two years since the public UPS had been retrained. So that's the gap between 3:00 and 4:00, just about two years, OK. We started measuring model performance decay. Well, as the first thing that we did before we started trading version four, we had a lot more resources for Version 4. We'd raise the seed round of funding. We had some data infrastructure there.

So we're like, OK, how much worse did it get over 2 years? And the answer was on median, something like 15%. Jay, correct me if I'm making numbers up something like that over 2 year decay, that's actually pretty good for any deterministic model. I was gonna say like what it what would you have expected? I honestly so, so a couple of data points is that 48,000 vulnerabilities were released this year. In 2021 when EPSL started there I think were 18,000 vulnerabilities were released. Yeah. So the growth is already like exponential, right? Just the exponent is not the same as it will be with these Frontier models.

Yeah, yeah. So I would have expected actually more of a decay because the system had changed and the volumes have changed, but OK. What's interesting is that the vulnerability and by the way, way more exploits too. Exploits used to be published and exploit DB and Metasploit. Now like overwhelmingly they're in GitHub randomly all over the place. So all of those things are growing exponentially. But the vulnerabilities that are being exploited, I think last year we tracked like 900 some and maybe the maybe 1100 or had active exhibition that we're not new. Yeah, yeah.

Overall, there's like 18,000 that have been and are being exploited. But of those 48,000 that were released that year, only about 1000 had exploitation activity. OK, yeah, yeah. So I think what we're gonna have to do and we are currently in the process of training at EPS version 5, which will account for some of this shift change. We'll make an announcement about it soon and then release it shortly thereafter after after sig, sig testing. But we're going to have to start retraining the model more often because the decay will probably accelerate until we get to another steady state where maybe there's some limit to the the frontier models performance. Yeah.

And before the decay was basically just, you know, the the market is shifting and it shifts slowly and that's what the decay is. But now I think that the underlying system is going to be shifting quicker. And so that's why that decay I think is going to be a lot more prominent. So retraining off it was going to be our key to dealing with the next generation of threats. Yeah, that next generation, I mean, it's, I don't know, the next like year to 18 months are going to be really interesting to watch. I mean, you mentioned that, you know, we used to think like, OK, attacker attention or time was, you know, a bounded resource that or bounding resource that they couldn't you can't really make more time.

You can't make more of yourself. But literally they can now they can sort of, you know, multiply themselves using AI. And on the on the defensive side, the scarcity of elite vulnerability research talent was a real problem too. You know, that there was a a very finite pool of folks during doing that top level research, you know, here in the US and around the world. And now it's just like, I don't know, do you have enough money to feed into the AI machine?

How many bugs do you want to find? Like so, money is a finite resource. Sometimes. Not always. Sometimes. Yeah. So I think this brings us to the point of why did Anthropic recommend a deterministic predictive model that's purpose built for one task and only does one thing when they literally build a model. They can do just about anything in a general purpose model. And the reason they recommended EPSS is it is a model that can measure that difference in attacker defender performance. And it is asymmetrically cheaper for the defender to run a predictive purpose build model for this task than it is for them to query a Jedi model every 30 minutes to figure out which vulnerability they should focus on next. So as defenders, we have some of these advantages where we can have sensor networks that give us real time telemetry about what attackers are doing and then use them feed small models that are super efficient and super cost effective. So they might have infinite

attention, but they don't have infinite money. And creating that kind of financial asymmetry is also useful. Yeah, that's a great point. I mean, yeah, the idea of making attacks more expensive in terms of money and time for attackers is always good. But we don't, I guess we have to assume that the apex adversaries that we're worried about all the time in terms of like state actors from China, North Korea and Russia are doing exactly the, you know, have their own labs doing this kind of work as well. Yeah.

You know, the bug finding and exploit development and all that kind of stuff. So yeah, yeah, I don't. Know, and the another thing I'll say like they're really high profile targets that are typically at the top of the list for state sponsored activity should be very afraid. And I'm sure that they are. But I think for those not at the top of the list for that group, I do want to point out that like the the number of targets at those places generally was not the limiting factor, right? I mean, like they had vulnerabilities. They have vulnerabilities, they will have vulnerabilities tomorrow.

You know, that that is not the limiting factor for most of the attacks. And so moving forward, there's going to be even more, you know, it's not going to change a ton. So they're they're going to probably have to update and react a little quicker, but hopefully we can, you know, get the prioritization better so that they can react smarter. So if Ed was on this podcast, he'd talk about the Mendoza Line. Yeah. The Mendoza Line for security is essentially you want to be better than an amateur and an amateur used to be.

Don't have anything that Metasploit can reach across your networks, across that data set of organizations that we worked with for the prioritization and prediction reports, I don't think there were any that didn't have at least one Metasploit exploitable vulnerabilities somewhere across their networks. This was 2021-2022. The thing that's really different is that the Mendoza line is shifting. It's no longer have nothing that's exploitable by Metasploit. It's now have nothing that is easily exploitable by an amateur wielding a cheap LLM.

So that line is shifting, but the the problem set and like the the constraints that organization has to deal with it really aren't because they were never above that Mendoza line that would existed in the past anyways. They had to manage to a different line, but the management process is still very much the same. Right. Yeah, In the past you were worried about, Yeah, somebody that got ahold of Metasploit and, you know, had some idea how to use it. One of the other things that kind of worries me now is these LLMS finding bugs in, you know, cloud platforms or SAS platforms that are used by everybody and their brother.

And, you know, some universal bug like that that you know, was found by an adversary and they get an exploit for it. Like there was a stupid C panel bug this this week that like effects 10s of millions of like shared hosting instance is and like the average enterprise that is affected and can't fix that, they need their hosting provider to fix it. So, you know, things like that, you know, those happen every once in a while, but I worry that those could be happening more and more often with, you know, mythos or whatever other frontier model decides to go and look.

Yeah, I agree and. Those are the things that enterprise, you can't really plan for those, you know. There's a whole class of those, right? Like it used to be the CV. It was something that the impetus was on the user of the software to remediate and patch the system. I'm writing Adobe Reader 91. I got to apply the patch on my endpoint. Increasingly, and this is anecdotal from just me reading CVS day in and day out, there are CVS in Salesforce or some cloud hosting provider that are there almost as like a record, but I can't do anything about it.

Nobody is using that software. The ownership of the remediation is on the provider of the software and that is not the same CVE as a CVE on a device where I have access and I'm responsible for that. Right. Yeah, those are those are the ones that I always, they catch my attention. You're like, those could be really bad. You know, when you sort of go and look at how many people have that, you know, that Salesforce, you know, they're using that cloud service or whatever it happens to be. Yeah, but.

At the same time, for the average enterprise, it's like, would I rather have my security team of three people fixing this or Amazon security fixing? Yeah, no, that's, I mean, that's one of the reasons you offload that responsibility to those folks, right? Like, yeah, Amazon and Microsoft have all the security talent in the world and all the money in the world. So hopefully they're doing, you know, they're spending our money wisely. Vulnerability tax. Yeah, yes, it comes for us all.

Yeah. All right, Guy. Anything we forgot to hit, I think we we got a lot of we got a lot of topics in there. Yeah, everything, I think. I I I really like anthropics recommendation. I think it is directionally correct in the sense that remediate the things you know have active exploitation that are relevant to you. You can't remediate the entire capitalist because you don't have every vulnerability on that capitalist. I think the caveat that Jay's research with the RE exploitation shows and that folks can read on the DBIR in the coming weeks is that just being on the capitalist is not right.

It is an indicator of risk, but it's not the only one, right? So recency of that exploitation really matters. I think we're trying to challenge the industry to think about the acronym Kev as not a binary but a continuous variable, right? Was it exploited yesterday? Is it still being exploited? What is it patchy? Is it bursty? And then the second recommendation is use a cheap, small purpose built probabilistic model to forecast what's next. And that recommendation coming from an LLM lab is really exciting to me. Like, you know, I don't ask Anthropic what the weather is. I still go to weather.com.

There's a reason for that. I, I had a really funny thought and I was wondering like, did they folks at Anthropic like use Claude to generate that blog and Claude just like shoved it in there and they never reviewed it and they're like, I whatever publish, you know, that seems right. It does seem right. Yeah, it'll be. Poetic. I think the chances of that are quite high, yeah. I don't remember if there were actual names on that blog, but it doesn't matter anyway. Doesn't matter. Yeah, not at all. All right, right. Well, guys, thanks so much for for coming on and explaining the world of math to me.

I really appreciate it. Yeah, great talking to you, Dennis. Yeah, great to see you guys. All right, take care all. Right. Bye bye.