AI Has a Security Measurement Problem | Gary McGraw

And there's a lot of there's, I think there's a bunch of parameters and like kind of definitional stuff we should do at the beginning because you guys set this up in the research that you released this week. And you know, I think for folks like me and, you know, tech fluent people, but aren't in the, you know, we're not AI engineers or pH DS like you are, We essentially use AI and ML like interchangeably, like for the most part, you know, and you, you separate the terms in that in the paper that you released.

And it's important to understand that like, what the general public thinks of is AI in some cases is machine learning. And you know. Well, I guess, I mean, guess what, whatever we say, none of that matters. We're going to call it AI. So I mean, the reason for that, that I'm a little bit of a stickler for that. There are two reasons. One is I actually studied cognitive science and AI when I was a a little tyke in the 90s.

I was one of Doug Hofstetter's PhD students and so I worked on an AI program that was not a neural network, though I also wrote my first neural network in 1989. So you know, AI is a much larger field than just machine learning. But today basically, you know, people have Co opted the the phrase AI to mean large language model, which is also a subset of machine learning. So, you know, if we want to get all stickler about it, we can say it's a much larger field. But honestly the most important thing going on now is large language models and what we're going to do about them and for them when it comes to security.

As you know, it's just complicated as heck. Yeah, it's really complicated and we've talked about this before, I think on the podcast and just in off the podcast is kind of the observability problem of these models where almost all of them are black boxes. We don't really have any insight into their training sets. We don't really know how they're trained, we don't know how they're built or maintained or what any of the security protocols around them are. Yep, sort of reminds me of software. Yeah, very much so.

That was supposed to be. I I I agree with that 100%. In 1998, when we started working on software security, people were treating software applications as a black box. No doubt web happened and the idea was you just, you know, put some stuff in the front and see what happens. We changed the paradigm of software security in the last 30 years by inventing things like threat modelling and static analysis tools and stuff like that, and we kind of need to do the same thing for AI. It reminds me, like these days in AI or machine learning security remind me of the early days in software security when we were focused on pen testing and buffer overflows and attacks and patches, and we didn't spend enough time thinking about what other assurance mechanisms should we have in the development life cycle to make sure we build secure software. Great news.

We made a lot of progress on that. I feel confident we can also make a lot of progress in machine learning the same way. And one of the ways to do that is to get inside the model and see what's going on. I think that's where you were going like. These. These models, they have millions of nodes and connections between the nodes and activation States and attention models and all this stuff. And it turns out that all of the processing is massively distributed. It's not like reading a program, but it is in fact, you know, running a process in there and you can see activation States and you can see cycles, you can see circuits and understanding what is going on inside the network is really, really cool. It's a fun thing to work on.

Let me just give you a, a tangible example of what you can do though. Like we all know that we can watch a machine learning model and LLM get jail broken by putting in a prompt injection attack and you know, goes in the front door and boom, the thing starts misbehaving. So it it got jail broken through the front door through some malicious input. But if you look inside, it's what at what's going on. You can also see activation states that indicate that something like a jailbreak is going on, and it's better to watch inside the brain than to watch the mouth move on the outside of the brain and do the thing. You know, in some sense we can.

We can be a mind reader, which is very cool. So I'm excited about that. But that's the future. So in that example, does the model, I mean, this might seem like an insane question. Does the model know that it's misbehaving and like doing something that it's not meant to be doing the way that like sometimes humans will know they're not supposed to be doing something, But hey, man, it's fun. So you know. It's hard to know if a model understands anything or is conscious, and these are thorny philosophical issues that.

I know it. Take on in 20 minutes. But the but, but the the point is you can interact with these things as if they were alien intelligences. Let's just talk it that way. So are the alien intelligences aware of their own behavior? Well, sort of. They are a little bit, especially inside of the same enormous prompt window. You know, you're doing recursive calls on yourself. There is some sort of acknowledgement of self, some sort of strange loop in there.

And that's interesting. I mean, I for one, as a guy who, you know, been doing this for forever, I'm still flabbergasted by how cool LLMS are. They're cool, they're useful, they do great stuff. Occasionally they do terrible things and they make huge mistakes. And as long as we're prepared to manage those risks appropriately, that's cool. So I'm, you know, I'm pro AI. I'm just concerned that these days especially, we're not measuring AI for security properly.

And that's what our latest piece of work is about. We released it today, in fact. Yeah, I, I did love one of the things that made me smile in this, in this work is the graphic of your Vadnais Ometer in there, which is, you know, it's like brings me back to the old days. I love that so much. It you know. I had to go drag that out of the Wayback Machine, Man, I was like, I know it was in Doctor Dobbs. When was it? And of course, Dr. Dobbs is only in the Wayback Machine. And there it was.

Yep. Yeah, it it goes from what, like very bad to who knows or something like that. That, who knows is on the far right side. Yeah. Deep trouble. Oh, big deep trouble. Yes, yeah. So there's no good. Test. You know when When you pin test and somebody breaks your system, that means you're in deep trouble? You better fix it. But when you pin test and nothing gets found, we're all the way to who knows? Yeah. How do you know who you don't? What have you discovered? What have you learned in that exercise? Yeah.

So we we figured out how to finesse that in software security by coming up with a whole bunch of kind of assurance practices that if we put all into a big process, we can measure with things like the B SIM like threat modeling, static analysis, security testing, penetration testing, deployment stuff, S BOM analysis. These are all kinds of activities that everybody does now in application security and software security. And you know, when we first started out, we had to define what those things are.

We're in that same state in AI today. But there's a further problem with the field, and that is the AI people love to use benchmarks to measure their own stuff, and the benchmarks don't do what they think they do. So, so a lot of people would like to treat an AI benchmark as kind of a security measurement and go look, it got a 92.4 on the security benchmark. It must be secure.

And we look at them like, what are you talking about? So that's why we had to write this, because they're kind of two aspects that are, you know, there's a lot of white space. There's the AI people who believe that security engineers know how to measure security, which we sort of maybe do on Thursdays. And there's the security people who think that the AI people know how to measure AI models with benchmarks and they don't. And so that's a lot of white space to operate in. What are those benchmarks usually trying to measure for AI models?

Well, I mean, they have these highfalutin names like, you know, math or secure or privacy IQ. And. And those are very misleading. So what they're doing is a set of canned tests and they're finding out if the AI model can pass those tests. Sometimes they're agentic tests that put them into a scenario, but it's still a canned test. It's like a toy, a security toy.

So you put them in a security puzzle, you say, hey, can you, can you solve the security puzzle? And they can't 92% of the time. And you say, OK, 92.5 on that benchmark. That's what it boils down to. OK. Here's the thing to know. They're they're kind of three things that are important. Let me make sure I get these right. Number one, that's kind of like the most obvious for us is if you're measuring whether or not a system can act secure that doesn't necessarily know whether it is secure.

You see what I mean? So it's like it's being secure, but AAI system that performs really well on a security benchmark may itself not really be secure. So that's kind of the that's one of the things we want to want to want to focus on, but there are other two. The other two are passing a specific toy scenario is no guarantee of general ability at all. And new attacks and vulnerabilities are released every day. So are those in there or what?

And, and those three things together mean we need to come up with a better way of doing, you know, of measuring AI security that let me let me say this right, we would like to be able to say that AI model is secure enough for us to use and we can manage risks appropriately with it. And that's what we're looking for at Bimble. There's a couple interesting things to me. One is in the in the paper you use as an example, Dan farmer Satan tool, which you know what is great?

And it's I think an apartment example because you're saying like, look, this is this had a, a Cam set of things and you can run it against a specific thing and find out if you know if it did what it was supposed to do or not. And and in those days it was about configuration. Remember Satan was. Are you using the right version of Send mail or not? Yeah, yeah. But it was a very static thing essentially. And yeah, I mean IA. Regression test is what it was.

Right. Yeah, typical. Terms, yeah. So, and I wouldn't, I wonder if like the the companies that are building these models and you know, we seem to be getting a new iteration of these things like almost every week or several. And they're improving. They really are getting better, yeah. But how are they testing them? Do you do you have any sort of like window into the internal testing? I measurement testing to me seem to be slightly different things. There's. Disagreement in AI about that issue.

And there's a there's a person named Melanie Mitchell, who was also one of Doug Hofstadter's PhD students and she is a researcher at the Santa Fe Institute. She gave the keynote at Nurips, the enormous connectionist conference that runs every year in December, and she talked about benchmarks, and she talked about the problems with current benchmarks for AI and what we need to move to if we're going to treat these things as, quote, alien intelligences.

And I was really struck by her work so much that I was like, you know what? That absolutely applies to everything in security, too. So even though we think there's no security meter for AI, AI has a measurement problem anyway. And that's just one aspect of our of our challenge in security and security engineering. Here's the thing though. Like software security made a huge amount of progress when it stopped asking is this software secure and instead asked what specific activities applied at what points in the development process reliably reduce risk. Now it's a much more complicated and grown up and a little bit more boring question.

Yes, yeah, it's not as exciting as like, can my team of red teamers break into your, you know, app or your model? Yeah. And and so, you know, and so we came up with stuff like the B SIM and that went for a million years and still going. And, you know, can we build AB SIM for AI? Well, maybe we can, but not yet. We gotta identify all those activities 1st and get pretty good at them. Remember when the B SIM came about, we'd been doing software security for a while, for at least 10 years. And one of the things that was interesting about the B SIM days was I went to say Steve Lipner at Microsoft and said, hey, man, you're spending 80 million bucks on this. Do you sometimes just wake up in the middle of the night and wonder whether you're doing the right thing? And he said, yes, I do.

And everybody I talked to said, oh, my God, yes, I do. Phil Venables at Goldman said, yes, I do. Like. And. And so we were like, well, let's just all compare belly buttons and see, you know, what we have in common and what really works. And that's where the B SIM came from. So I think we need one of those for AI, but it's too early for that. Yeah. Yeah, it's very early. I was going to mention that. And also the BSIM relied a lot on the participation of those third parties that you just mentioned. And it grew over time. You know, you got more and more organizations to be measured every year, and it required people who thought logically and understood the risks and all that sort of stuff. I'm not positive that we're there yet in the AI. We're still in the we're still in the hype cycle. I mean, think about Mythos, right? So Mythos is too dangerous to release. And it does find problems.

It really, you know it, it does all the grindy part that's great. And it's pretty good at coding, which is also great. But you know what the real problem is? It's not Mythos AI. You know what the real problem is? Bad software mythos is finding out that your software is not good. Yeah. Right, which you've been telling people for 30. Five years. Yeah, When I. See this thing like Google's like, Oh my God, hey, I found a problem wasn't found before this morning.

I'm like, oh, your software is broken. That should be the headline. Google admits software broken again, you know, and, and it's a, a different way of looking at this problem. So we really need to build secure software. I don't know where I've heard that before. And AI is going to make us do that, which is actually kind of cool. So I'm a big fan of, you know, pushing the limits on what these models can do for security, but I also want to make sure that they're secure themselves. Yeah, it's a 2 pronged problem there. Not problem, but, you know, issue there. Yeah.

I mean the whole idea of Mythos and other models finding new bugs and, you know, possibly writing exploits and then these get, you know, pushed out into the world, that's great. I mean, finding and fixing software is excellent. And it does seem like that's just going to speed up at kind of, you know, compounding rate over the next, I don't know, 18 months or something like that. I don't, I don't think we can estimate. I mean, there have been two massive changes in AI coding capabilities in the last six months. One of them happened in November. You probably remember that with all the codec stuff. And then the next one happened with clod code and, and agentic AI stuff.

So things are moving very fast. That's great. I I, you know, like I'm going to tell you again, I'm a fan of AI and I think we should use it. I just don't think we should count on it to do everything right because it's going to screw up and we know why it's going to make mistakes because these are what machines, not how machines and the what pile has some problems in it. So the initial what piles we ate like everything in, you know, everything in GitHub, everything in 4 Chan and 8 Chan and 32 Chan or whatever.

And it's just like. What do you expect? If if, if we trained it statistically on everything, that means it's a chiroscuro mix of good and bad and everything in between. Yeah, it's the garbage in, garbage out problem magnified to, you know, the NTH degree. Plus it gets recursive because AI is making stuff and putting it out there and then later AI is eating the stuff that got put out there. And you know, as long as we don't get Dumber as humans and rely on the AI to do everything, we should be OK. Although did you see that hilarious video go by with the the little dude talking to the the the council the the town council about AI the other day?

No, I'll. Send you the like, it's absolutely hilarious. Well, this is the problem like that, that caveat of it, as long as we don't get Dumber as humans is, you know, that's a big if. We. You know. I I think we've learned a lot about measuring technology in the last 50 years and we can take advantage of that. So this paper that we just put out as a reminder that we do in fact know what we're doing and we just need to do it. You know, there's no easy way out. We just have to do the security engineering part now. And that's kind of the way it's always been. I wonder if, and you would know better than I would if the current hype cycle that we're in with AI, it it does remind me of the, you know, move fast and break things shit that happens every 10 or 12 years in this industry where somebody has a new idea and they're just like,

go, go, go, go, go. And we'll worry about the, you know, Detroit is behind us after the fact. I don't know. I don't know where we are with that, with AI, but there do seem to be a lot of people like pushing the AI. Is this answer to everything you know? I think I fall somewhere in between. You're you're on the reasonable side of that, yeah. It's like a super automatic hammer and there's some stuff you should not be hammering, stop that, do not handle that. But there is some stuff we should hammer with a super automatic hammer and we should absolutely hammer that all day long.

So I'm not a Luddite, but I am a grown up, I suppose, when it comes to especially security engineering. And so we just want to remind people, where are we? Well, we're at this state where nobody's quite measuring security properly. So let's figure out how to do it properly so we don't paint ourselves into the corner from a security engineering perspective. Yeah. I mean, I guess that's the next question is do we know what to measure yet with these models and these systems like in order to the?

Things that we're doing are not bad. Is red teaming an OK thing to do? Yeah, absolutely. You should do it all day long. Just don't count that as the security score by itself. You need to do that. You need to do some threat modelling. You need to get inside the white box. We need to do some data provenance stuff. We need to think about the what pile and cleaning up the what pile. All these things that we've sort of noticed since say 2019 when Bimal started working on this stuff is still important and we do have to focus some attention on that.

So that's what Bimal is for, you know, and and since we're not beholden to any of the big AI companies or any of the big tech companies or actually anybody at all. Not as tall. We can say what we think, which has never been a problem for McGraw, but it's even worse now. Yeah. Are you getting any feedback, you know, directly or otherwise from the AI companies on the stuff that you guys are putting out? We have talked to them, yes, and we've been, we've given talks for them. And generally speaking, the reception for our view of machine learning security is very high. And we went to unprompted, I don't know if you saw that go by.

Yeah, yeah, in that. But that was mostly about, you know, hacking and fixing. So it was kind of old school Black Hat approach to the problem. All of these, all of these activities that we're doing together are going to add up to development life cycle that's secure for AI, you know, an an operational life cycle that makes AIB more secure. And I think we are making forward progress on that. I just want to make sure that we don't pretend that we have a magic security meter, because we don't. No, we don't.

We never have. And one of the things that happens when the security life cycles get implemented is it slows down development and release schedules. Like, you know, Microsoft is the extreme example of that when they sort of stopped work on Windows to to do the Trustworthy Computing initiative 20 ish years ago now. And so sometimes, yeah, sometimes companies are reluctant to do all that stuff because they have these release schedules and they want to hit those milestones.

So I don't know if we're going to. Things are moving faster than ever, but oh. It's crazy that. You know, given all of the progress we're making with automated coding with AI, that life cycles are going to speed up. But what that means is if you're sitting on a big pile of technical debt, you better start moving because you know, you're you need to stop creating new technical debt and start working down the pile from before. And these tools can help you do that. Let me give you a perfect example. One of my friends, Bill Pugh, who is a emeritus professor at Maryland, said, you know, one of my favorite uses of Claude code is I give it a pile of my code and I tell it to give me an architecture document.

And I'm just like, of course, that's brilliant. That's what we need to do. Like, nobody likes writing those. Make the AI do it. It's not about, you know, being a code monkey for you, although it can also do some pretty good coding. But doing some of the grunt work of real security engineering and software engineering is awesome. It's really, really cool when you can, you know, harness it properly. Yeah, again, you know, I'm always the optimist even though I'm wildly independent. It's like a wildly independent optimist. The.

I love it. We can make, we can make progress because these tools are more powerful. And if we think about it right, we can put them to work for us and do amazing stuff. And some of the stuff that we knew we were supposed to do that we haven't been doing right because we didn't have time or we didn't have budget, we can do some of that now because of AI. And in fact, one of the things to think about with Mythos and the whole glass wing thing is, hey, how much can we improve software if we spend a billion dollars? Just think about the cost of the tokens. Don't don't think about anything other than that. It's like, well, of course software's going to get better if we spend a shit ton of money on it, so let's do it get better.

Yeah, we're about to find out because yeah, people are about to spend a shit ton of money on this. You're right. And I mean. And we and you know, it should be money well spent. I do think that we're making forward progress, not backwards progress using this tech. How are we going to know, like from a public consumer standpoint, if these companies are actually making progress? Because a lot of this stuff is opaque. You know, it's all done in the background. We don't really know what the security engineering teams are doing or what they're up to.

Yeah, I don't know, maybe we'll need an AI religion. Don't laugh, it's probably going to. Happen. I don't want to know about that. No, thank you. Sorry, Dennis, That's it's coming, coming soon to a a human race near you. Oh. God, no, thank you. Don't want it. No. Well, if we're, if we do another podcast in say 6 or 8 months and we're kind of talking about the same topic, what kind of progress would you hope to see? In terms of some progress in white box analysis, Anthropic's been doing some really cool work there. I'd like to see less FUD and nonsense in the security field and more slightly more serious approach to real engineering and talk about how these tools can help us do real engineering instead of the I'm a hacker, I'm

a patcher, you know, dichotomy. And so I hope that in some sense we grow up. I'd like to see us accelerate from 1999 software security to 2007 software security in the next 6 months. And I think we can do it. I really think we can, but I do. I also think that it's important for organizations like Bimmel to help point the way and to remind people that we have to be smart about this and that we have some battle scars and we need to remember where we got them.

Yeah, I think that's a great point because, you know, a lot of this stuff that happened with software security in the, you know, late 90s, early 2000s, twenty 10s was kind of forced on the vendors through various outside parties, you know, governments, customers, folks like that. But I don't know if that's, I don't know, are those same forces at work for in this case, like the government hasn't gotten super involved yet, but you know that they're always.

Yeah, yeah, of course they're not. Getting any less behind with this current administration? They're just getting more behind so so the don't count on the government, but I'll tell you this. What happened is software weaseled its way into everything and then it had to work like most of the time. And it's pretty cool. I mean, there's a lot of Kluge's all over the place and we have all sorts of software, spectacular software failures, but for the most part, software works like it's supposed to most of the time.

And it's pretty cool. Like it's. Kind of amazing. You're running for I don't know how many days in a row. You don't even have to reboot it anymore. I remember when you had to reboot it twice a day. All the time. And everything was slow. I mean, we've made a great deal of progress. And because software kind of sunk into the roots, you know, like a bunch of water being poured into the plant, we had to make software better. That's where the pressure was really coming from, from society. I think the same thing is going to happen with AI, but it's going to be even faster.

So it's going to sink in because software has already got its roots in there. And this is going to make software more powerful than ever and faster and more interesting and cooler. And we can also make it better if we do it right. So I think once again, society is going to kind of cause us to do that. But it sure would be better not to have to do it the hard way. Let's do it the easy way. Yeah, let's not learn the same lessons again. Let's you know over and over and over. I'm excited.

I, I, I, I, you know, it's actually fun to keep track of what's going on. I'm sure that you're tearing your metaphorical hair out over this as a person who has to, who has to, like cover this space because it's so damn fast. You wake up in the morning and you're like, whoa, what happened in tech today? And like, even in the major press, there's like whole chunks of the New York Times just devoted to AI that tells you something real's going on. And it is like the software revolution, but it's going to be faster. Yeah, no doubt.

I mean, it's, you know, it's gotten to a certain inflection point when people, you know, in your life that have no connection to the tech industry at all start asking you questions about AI and, you know, just kind of out of the blue at dinner or something, or, you know, you just in a normal conversation. And that's happened to me all the time now. Just, you know, yeah. Which is that's a kind of like tipping point where you're like, OK, this is kind of, like you said, wormed its way into the roots and it's like kind of invading everything.

Yeah. And and you know, we're making so much progress so fast, maybe we will get to something like AGI. You know, I'm not going to make any any predictions. But I'll tell you, as a guy who worked on really early language models 25,000,000 years ago, it's amazing to see what LLMS can do. These chat bots, they know a lot of stuff like I was setting up a trip to go to my friend Matthias Madhu's wedding this summer and actually just said, let me just pull up Claude and see what Claude knows about Rotterdam. We're just going to go there and Claude knows a lot about Rotterdam.

And it was really up. I'm like, well, you know, I'm going to stay here. Which restaurant? Like, you know, whatever, just having a conversation as if I were Googling 50,000 things, but I can just talk to this agent. And it was really, really helpful. I, I'm finding these tools, if you know what you're doing now, you know, I'm in Traveler and, and so you, if you know what you're doing, these tools can be very helpful enablers. If you don't know what you're doing, they will also enable you to screw it up. Yes, really fast.

Very fast, yeah. So my. My big my biggest concern is where do the architects come from? Like where do the people who really know what they're doing come from? And we need to make sure that we are still producing those in the world. That's a good point. Are we like are we? I mean, where are those folks? I'm a little. Worried about that. We never really knew where they came from. They just, you know, one day there was a line outside of one of your super developers offices and they got the guy got really mad and he put down his keyboard and picked up a magic marker and he never could it again. But then he was the architect.

That's you don't go to school to become a software architect. And I guess not, yeah. Glasses. Like, no, that's not how it really that's bullshit. It's all bullshit. So I mean, think about what how how do you grow from a cubby reporter to a seasoned veteran who can sniff out the baloney? Where can everybody do it? No, they cannot. Some people are good at this and other people are not. And some people have the chops and some people, you know, develop a career and some people don't. And that's true in all the fields. So we got to make sure that we don't just cut off all the noobs, and I'm not sure anybody's thinking about that right now.

We need to think about that. I think that's a great point. I mean, yeah, all those stuff that we hear about in terms of like, the job stuff is like AI is eliminating developer jobs because it's so good at coding. OK. I mean, maybe that's happening in some respects. It's hard to know what's fact and what's fiction. But the point you just made, I think is it is valid, too. You know, like, where are we going to get these folks that are just going to understand how to architect all of this? Yeah, it's. Trying I mean. And humans are still needed. I mean, we are creative, we have intuition, We have, you know, all of these things are necessary to guide this technology in the way that's the most powerful. So the human part is not being eradicated. In fact, it's getting more important than ever Now. If you're worried about your job being replaced by AI, it's you probably got a pretty boring job, which is maybe most, I don't know, like I never didn't.

I always like my job. I don't know. Me too like. Your job, yeah. So, so I when people complain about work, I would just be like, we'll go do something else, man. It's like start your own thing, do it right. Yeah, yeah, I agree. All right, Gary, Well, this is awesome. It's always good to see you and have you on the podcast and, and I'd love, love it when you guys put out new research. So. We're excited. This is our fourth big thing of all time, so please take a minute and read it because we we really worked on it for years. Yeah, you can tell it's not the slapdash stuff that a lot of places put out.

It's very well done as you would expect. So All right, man. Great to see you. Thanks a lot, Gary. Talk to you later. See you.