The Vercel Intrusion and What is Happening at CISA

These are the times we live in. They're just AI companies that you've never heard of. Every single day they put up their own security advisory saying in so many words, we were the compromise third party app. And it turns out that apparently one single for sale employee had used this app, this AI app internally that led to some Google Workspace shenanigans on the Versal network, which led to some, you know, the, the rest of the consequences. And for people that don't know Versal, it's just a big platform that a lot of developers use it.

And as such, it holds a lot of secrets, you know, secrets meaning, you know, API keys and things like that, not like national secrets, but secrets in the security sense. So people got very, very worried when this first came to light on Sunday. And there was a lot of talk among our security researcher friends and developers and folks like that about what the downstream impacts of this could be. And I haven't seen too much in that, in that realm yet. I haven't seen too many other companies or individual developers coming out and saying shit, this got me or anything like that.

It was more a bunch of sort of proactive, hey, everybody rotate your keys, you know, check your logs for suspicious activity. Here's what to look for, that sort of thing. But it was one of those incidents that moved really quickly and also just kind of shows how many of these platforms there are out there that if they get hit, you know, that obviously this isn't Cloudflare or, you know, it's not AWS or it's not Azure. But if something like this happens to a platform that's used by a big chunk of people, the effects are, you know, potentially really, really scary. Yeah, I agree.

It it seemed like at first everyone was kind of holding their breath with with this platform just because of that proliferation potentially of how you know what it could impact. But and the other thing too that sticks out to me about this incident, but really like a lot of the incidents that we've seen over the past however many months. Is that just the basically because all these platforms are so dependent on each other or like so many employees use unauthorized third party services or things like that?

Like when the information was coming to light about this incident, it was pretty funny. Like first we heard from Versal in that brief security advisory, then we heard from context dot AI about what their perspective was and how, you know, because of it was an upstream, you know, reach that went down towards them. And then we also heard someone else came out and said, hey, there was like a nympho stealer that was involved in the, like the, I guess there's like a Loomis stealer infection that was part of all this too.

So there's so many different components there and so many different ways that, you know, threat actors are able to kind of get through to, you know, finally end up in that one single instance of this employee that it's just like, there's a lot there. There's a ton, and I do think that this is I, I don't know that this is 1 of this isn't related to AI in general to me, but the fact that this was an AI tool that was, you know, sort of the root cause of this in general doesn't really matter to me.

What matters is you have an employee, you know, what looks like downloading and using a tool that maybe wasn't, you know, maybe their IT department wasn't totally aware of or hadn't sanctioned, which is something that happens at every company across the world every single day. You know, no matter how tight your controls are and how tight your policies are and how conscientious your users are, everybody does that stuff all the time.

You know, everybody is using some unapproved AI tool or, you know, productivity tool or podcasting app or whatever it happens to be. And this is, and, you know, threat actors know that they're just like, look, if I can, if I can find a way to to get my clause into, you know, even a relatively popular app or tool, it's, you know, Katie bar the door, who knows what. Yeah, Yeah.

So, yeah, I wanted to read a little bit of there was that initial security advisory that I mentioned that came out Sunday afternoon. And then they, they've been updating the advisory sort of as as they go. You know it, the advisory is good. Now there's a lot of information in it and they've, they've, unlike some vendors, they're saying, here's what we've updated this time. So there's a good timeline there that you can look at. But last night, one of their executives, I believe the CEO posted a, a sort of a long thing on Twitter about the current state of the investigation.

I'm just going to read a little bit of it. It says the team performed an in depth analysis to search for root causes better understand the behavior of the threat actor. We cast a very wide net pulling and processing nearly a petabyte of logs. That's a lot of the entire versal network and API extending well beyond the initial context dot AI compromise. We now understand that the threat actor has been active beyond that startups compromise threat.

Intel points to the distribution of malware to computers in search of valuable tokens like keys to Vercel accounts in other providers. Once the attacker gets a hold of those keys, our logs show repeated pattern of rapid, comprehensive API usage with a focus on enumeration of non sensitive environment variables. There's a lot of words that kind of, you know, detail exactly what you just said. Lynn's like they, you know, there was malware.

The malware went to places it shouldn't have been and the attackers were looking for sensitive stuff, which is what attackers do. But, you know, kudos to them for continuing to update this and being public about it. You know, once things like this happen, that's the best you can do is just be transparent and you know, upfront about what's happening and what you're finding as you go. You know, we've seen plenty of companies that didn't do that and they just get kicked in the teeth online, which is, you know, it's this happens, you know, doesn't happen to every company, but it happens often enough that just like learn from everybody else's mistakes. Just be honest about what's happened and go from there. You know, customers will be mad, but there's, you know, once it's happened, all you can do is move forward and try, you know, try and do your best.

Yeah, yeah. I mean, I think they've been pretty transparent and good, pretty good about updating. I wish we had more details about kind of the impact net in terms of yeah, customers, but I'm sure, you know, they're reaching out to customers and everything or impacted people with kind of what they're they're finding, which is good. So, yeah, yeah. Go ahead. Yeah, they did say that they've notified every affected customer at this point. So, you know, that does lead me to believe that it wasn't a massive number of customers if they're, you know, notifying everybody.

I I know you can send bulk emails. I'm aware, but you know, I don't, it doesn't sound to me like this was in the 10s of thousands of customers. It's hard to know. I know that it's a widely used platform, but it's hard to know how many. I I don't know if they'll divulge that at some point. I kind of doubt it, but. Yeah, yeah. One thing that I saw on Sunday when like the, you know, everyone was starting to tweet about this and distribute information was I saw someone tweet a picture of the alleged person employee that was part of this that was allegedly behind this breach on Twitter and that that account took it down.

But I was like that is not OK. What are we doing? I didn't see that. Yeah, that account came under some Flack for that. But then there was also a little bit of a discussion. And I think it's worth, like, highlighting just about, you know, that's not how we treat people that make mistakes like that, like it needs to. Yeah. What matters more is kind of the defenses that companies put in in place.

Yeah. And then also, we just don't do that. That's no. How security? Is Christ, I don't think I've ever seen that happen before. Yeah, at least not that I can remember. I've seen lots of instances of researchers or other people like tweeting supposed pictures of threat actors and things like that, like ransomware gangs on a yacht in, you know, the Mediterranean and things like that. I'm all for that.

But this kind of victim shaming like that, Get out of here with the man. That's rough. I. It was bad, Yeah. I was, I, I never like, dabble in that Twitter drama, but I was like ready to subtweet that. Then they took it down and I was like, you know, I'm not going to put more attention to that. Like because a, how did that, how do you know that quickly unless you're in the company and you happen to know who it was at that point? Yikes.

No, thank you. God, there there was a lot of like weird information going around at the time because right away, I think it was the very first Twitter post that had any information on this at all. Had a screenshot from one of the from Underground forums where this first came to light and somebody posing as shiny hunters had claimed responsibility for it and said they were selling, you know, the data in the access. And it turned out to be like shiny hunters imposters, which is the thing that happened.

She was just crazy because I don't know, some researcher contacted somebody that is in shiny Hunters, I guess and said, hey, did you guys do this? And they were like what? No, that's not us. And which is, you know, you wouldn't typically discount something that criminals say, But when you're talking about a group that is very loudly proclaiming that, you know, will typically say, look what we just did. If they're saying no, no, no, that wasn't us. I don't know what you're talking about. I don't know.

I, I, I tend to kind of lend some credence to that, especially when it there, there didn't seem to be any follow up at all from all of that. It just seemed like somebody was trying to grab some attention. You know, yeah, I actually I think that's why this this like imposter shiny hunters group like posted that and like there might I wonder if like this account might have taken some information from that. But to your point about the shiny hunters fake group, did you see the I don't know if this is the same group or a different one. Someone else claiming to be shiny hunters claim that they breached mythos and that yeah, but they came out with like screenshots and like immediately everyone was like, huh, these screenshots are a little suspicious. There's like these look like these might have been some vibe coded types of things.

God. Well. There's just a lot of misinformation running around. So God, don't believe everything you read on the Internet, kids. Geez. I. Know, yeah. So to recap, we're at the point in human history where people are pretending to have gotten access to an AI tool and are posting fake screenshots of their fake access to this tool, which happens to be named Mythos, which makes it sound fake to begin with.

I mean, we know it's real, but. A lot of this up, yeah. No, it's so funny to me. We a lot of times we talk in this podcast about how we know when something is sort of broken out of the infosec world, when you know, one of our non tech friends or relatives asked us about it. And I told you this a couple days ago, but I was playing golf this weekend with somebody who I know and doesn't work in the tech industry at all. It has absolutely no connection to the tech industry, works in the golf industry, so pays no attention to cybersecurity whatsoever.

And you know, halfway through the round, he asked me. He was like, so what's going on with this Claude Mythos thing? Is that shit real? And I was like, what? Where did you, Did you just like get some talking points on your phone or something that you wanted to ask me while we're like playing golf? Probably like whiplash of just like. Honestly, the. M. Word I I truly was like, who just asked that question? And I'm just out here in nature trying to enjoy golf. I don't want to talk about this right now, but it was he had some like pretty pointed questions about his first question was literally like, is this real? Did this does this exist?

And he knew out about Project Glasswing and all of that. And I said, yeah, I'm sorry to tell you it is real, but yes. And but he had lots of questions about, like, AI usage and what it's doing to the environment and how this is going to affect jobs in cybersecurity and elsewhere. And, you know, I didn't have any happy news for him yet on that respect. But it was just one of those weird, you know, cognitive dissonance moments where you're like, somebody that works for a golf magazine is just asking me about AI in cybersecurity research.

Yeah. We can't escape it, Lance. It's it's everywhere. I know I like in 10 years down the road, do you think that well, like people will look back at the announcement of mythos and like it could be compared to like, I don't know, what's like some of the the biggest things that you remember over the past like. I mean, yeah, things like, you know, on the malware side, things like the Code Red Worm or Sequel Slammer or things like that, or yeah, the rise of like large scale bot Nets or things like that. But those all happened over a pretty a relatively gradual period of time compared to what we're seeing now with this. But I do think that this specific moment in time like this last probably like 3 to 4 weeks that this has all been happening will probably be, you know, sort of looked at as like the beginning of something.

I don't know what, you know, whether that's going to be a positive or a negative sign. But I was in a, you know, a signal chat with, with some of our friends and on the research side of things and, and a few CSO's in there too. And they were essentially saying, you know, whatever you think about Mythos's capabilities, you know, real or hype or whether a lot of it is marketing, which, you know, I, I tend to think there's a healthy dose of that in there. But that's what companies do. It's probably, you know, the leading edge of a large wave that we're about to see.

Because you're just, you're seeing too many smart people saying that and using it in using Mythos and other frontier models in ways that literally were not possible three months ago. It hadn't really even been considered in a large way three months ago. So yeah, I, I do think that this is probably some sort of, you know, the beginning of something, whatever that, whatever that ends up being. Yeah, I know I and like, I don't want to go down the mythos rabbit hole again, but like, I feel like the when you think of AI, like, I don't know, I think, I mean, we talked about this even the beginning of this podcast, but like there's there's people are just trying to figure it out on in businesses and everywhere else.

Like, you know, you've got businesses with the shadow AI problems, just like we just saw with Versailles. And then on the other side of the coin, like I think threat actors are probably and you know, we've seen this with many other things outside of AI, but threat actors are able to move fast, move quick and in how they. In their operation. So like, I'd imagine there's probably some behind the scenes like organization happening in terms of like how they're finding out how to use these tools and sharing that information and things like that. No doubt.

Yeah. So yeah. We'll and yeah, I mean, whether they have their own, you know, I'm quite sure that Russian state actors and Chinese state actors and I don't know, maybe North Korea, I kind of doubt it. But that level of threat actor have access to their own extremely leading edge AI models or LLMS that they're building their own on their own. Or, you know, maybe they have covert access to some of the some of that, you know, the anthropic and open AI have no idea, but you can bet they're trying. But yeah, I, I will be interested to see, you know, I'm sure we're about to find out how threat actors are, are using these things to tune their attacks and that kind of stuff. I think we've seen it to a limited degree in the way that they're automating the

development of, you know, fishing kits and fishing lures and you know, some of the the voice scam campaigns that we see that sort of stuff. But I don't, we haven't seen a breakthrough in a big way yet, I don't think. But I'm sure by this summer we'll all be talking about it at Blackhat or something of, you know, what's been going on in the last three months. Yeah, yeah, hopefully. But yeah, And I if we are talking about mythos, I do need to mention the unauthorized access thing.

Yeah. That was, yeah, I feel like that was just something where people saw that and were like, OK, this is the most cybersecurity esque story that I've ever seen. It was it was the most like infosec drama ass story that I've seen in a long time. Yeah, yeah, it's, I don't want to take into that too much, but that some of the headlines I saw were some of the worst headlines I've seen in a long time, like click bait, scare tactic nonsense headlines that you just like, come on. And then there's like 60 words in the story, you're like what the fuck?

They're screenshotting like a discord post or something. Oh God, that was. It was classic. Yeah, it truly was. Yeah. I hated every minute of that. I tried to pull the RIP cord on that real quick because you could tell that it was going to go down the drain, like very fast. I've seen way too much of that. Very much so. Yeah, I did want to. I want to circle back. Let's get off the the mythos thing for a second, like 3 weeks ago. Let me see if I can find when exactly this was. April.

Yeah, three weeks ago, April. First there was the big story about the Axios NPN supply chain attack, which, you know, did have a lot of downstream effects. Very big story. We wrote about it. We talked about it a bunch of times on the podcast. Turned out to be a big thing. Even though the threat actor only had access to the the maintainer's environment for, I think what turned out to be less than three hours. It was like a very short window. Everybody can go back and read the stories to get all the details, but I, at the time you and I sort of brought up the fact that Sisa never said a word about this. There was no alert about it.

There was no advisory. There wasn't even a, a tweet or anything from Sisa on this pretty large incident in the ecosystem, which I would imagine probably had some effects in the federal government too. I, I would bet a lot on that and never heard anything from them. And then two days ago, I think SISA put out an alert about this. And yeah, OK, three weeks later, they just put it on alert with no context of like, hey guys, sorry we're late.

But just like here's something that happened that we're aware of in here, a bunch of details, A bunch of people lost their minds 'cause they, they thought it was another one. They thought it was a new incident affecting Axios and you know, now I, I even did a double take. I was like, wait a second, is this a new one? Or, you know, this is a thing that you go through sometimes with like 0 day and like edge security appliance where you're like, is this can't possibly be another one. And it is, but in this case it was not.

But sis is just putting this out like eating it out into the world on like the middle of a day and just be like, Hey, everybody looks like we caught up. What do you guys think about this? I, I couldn't believe it. I you know. And the way that the way they put it out too was it was a tweet. It was very ambiguous about when it was. So there was a tweet and it was like this little like breaking news siren emoji, like one of those like images that had I think it was like it was a big thing that said like alert and. Like, yeah, it had like the full on social media card of like, you know, the bullhorn and like, the whole thing of like this is happening.

And so I took the bait. I looked at it. I was like, no way, no way, no way. And Nope, it was 3 weeks, just three weeks late. So. So yeah, giving people heart attacks, yeah. Yeah, yeah. Yeah. This happens occasionally or not occasionally. It happens all the time when they add a bug to the Kev, which is like, that's understandable that like that can happen weeks, months, years after the, the vulnerability disclosure, right? And but with something like this, that isn't a Kev entry. It's not, it wasn't even one of those like joint advisories that they put out with NSA and the UK and Australian governments where they're like, here's some remediation guidance or here's a snapshot of what all of these intelligence services are seeing on the threat landscape.

It was it was literally just like they turn in their homework three weeks late. We're like, is this good? I don't know. Yeah, that was rough. Yeah. I didn't love it the. Comments were pretty funny, Like someone tweeted this is like warning about a tornado three weeks after the fact. Someone else said this is not LinkedIn. You're several weeks late. This is not LinkedIn. I'm thrilled to announce that we've caught up and honored.

And yeah. Oh. Man, that's great. On a serious note, like it is like, I don't know, it's kind of sad. Like just like I mean they there is some major like budget and resource things going on. It's so sad. And like other parts of the yes, obviously, but you know, like we, I feel like we got to see over the past, like however many years, like just how like we saw Sissel really be built up into this, this agency that had a lot more control over everything.

And so it's it's kind of sad to see this. Now, yeah, that's the larger. The larger point is that we SIS has been gutted in the last 14 months, which is extremely sad to see because we saw what it could do, you know, when it had resources and you know, the money and people and smart people leading it. And now we're seeing the other version of that. And you know, I know that the people who are still there are trying really hard and are doing a pretty thankless job most of the time.

But when you reduce the staff by, I don't know, we'd have to go back and do some math, but by a lot and a lot of the senior leadership leaves. You're this is, you know, things fall through the cracks. This is what happens. And, you know, this didn't turn out to be a national crisis, but that was a pretty big supply chain attack. And you know, there's still there's still things happening from that that wasn't a minor, you know, malware campaign.

That was a big deal. Like imagine, like remember when solar winds happened? Like since it was pretty on top of that and we had a lot of kind of advisory type of material from them with guidance and like, here's what's happening, here's you know what, what we need to do right now. And like, imagine if there's another, you know, event like that, how you know companies are going to be looking for.

You know. Some sort of help from the from the government. I don't think they're going to be able to get that, let alone the fact that, like, you know, a lot of federal agencies depend on, you know, some of the resources that CISA has to offer, so. Yeah, they all do. Yeah. They, they rely on them for instant response and investigation and all of that kind of stuff. And they're pretty diminished at this point. So they're their capacity to provide that stuff is not it's not what it was, you know, which is sad to see.

I don't want to harp on it, but it was just seeing that that advisory three weeks late was just it was kind of like a, you know, wake up call. Not not for us, but I mean, we pay attention to this stuff on a daily basis. But you just kind of point out it like this is what I'm talking about. Like what what are we doing? Like, this is just not, this is not how we're supposed to be doing it. Yeah. But yeah, I did see they had a an advisory come out today.

I mean it was in conjunction with a bunch of other agencies globally. And I'm sure that's it's kind of been, you know, something that's investigate being investigated for a couple months. But this thing about China Nexus covert networks of compromised devices, looking specifically at, you know, compromise small office, Home Office routers, IoT devices and how Chinese threat actors are basically using these to create these like networks and that are that can be like constantly updated and all these different things.

So, so I saw that and I was like, OK, there's there's some sort of like. Yeah, they're still doing stuff. Yeah. Yeah, it that's a good one too. I've only read about half of it, but that's what you want to see from them is here's kind of a novel thing. You know, it's not completely people understand that advanced threat actors use IoT and Soho routers and things like that whenever they can because they're usually hard to update or never updated or impossible to update or whatever.

And once you get on one, you're just like home free. But it was a pretty detailed advisory of like China state backed actors are using these as their main infrastructure in a lot of cases. And it's like shared infrastructure among these groups, which, you know, it lends another is kind of another level or layer of, of the complexity to the China problem. I think specifically is just how much those groups share TTPS and infrastructure and knowledge and all of that kind of stuff. And just how dedicated and like efficient they are. You know, they those groups understand what their assignment is and they, you know, they go and get it. Yeah.

And they know what types of devices like, you know, can be easily compromised or things like that. They understand kind of that that attack factor really well. Yeah, if you want to like, be like scared and depressed and just like annoyed sometime, go and read like an IoT report, like read anything having to do with like Mirai or any of those IoT bot Nets, or go and like dig through Gray noises data on some of that kind of stuff. You'll just be like, oh, OK.

Literally like 90% of the TV's and appliances in in America have some sort of, you know, rootkit on them at this point. But OK, I guess, yeah. I don't want to be scared or depressed. And now I know I Yeah, sorry, we don't need any more. I know. We really don't. We like to bring you guys good news, but this isn't that week, unfortunately. I know, yeah. I was trying to think like I was like, can we end this podcast with some sort of good news? And I really cannot think of any sort.

Of no, any sort. I can't either. I Yeah. And we're both generally optimistic and, you know, upbeat people. But this is it's been a bad week for optimists, unfortunately. Yeah, Yeah. Well, there's always. Next week? Yeah, there's always next week. Who knows what you know, fresh horrors. Oh, wait, on the other side of the weekend. You never know. You never know. Even a good news isn't you might. You might, right? Yeah, Well, I can't even escape from it on the golf course. Who knows?

It's crazy. Then even like a couple weeks ago, I was on FaceTime with my daughter at college and one of her friends, who I know pretty well jumped in to ask me like AAI question. And I was just like, I I no, no, I'm not. I don't want to talk to you guys about AI. Just stop and don't and don't want to do this. I want to hear about you guys, like going out to the bars and having fun. I don't want to talk to you about AI.

Yeah. I'm sorry there needs to be like a like a safe word for AI like like get away. Yeah, yeah. Like just maybe that's our next T-shirt. Like please don't ask me about AI. Yeah. Yeah, I will not talk to you about it. I refuse. But on the, I guess on the good news side, we do have, we've got, we've been threatening this for a while, but we do have the podcast schedule with, with your colleague John Hammond to, to do Black Hat the movie, which if I think we're doing that about four weeks from now as we're recording this. So like a month out, that gives everybody plenty of time. Go and watch this absolutely bonkers movie. It's on Amazon Prime.

I don't know if it's free right now. Sometimes it is. I discovered a couple weeks ago that I at some point purchased this movie from Amazon Prime. So. Really. Yeah, Yep. Because I went to watch it and it was like, you own this like I do. When did I pay $20.00 for Blackhat? But apparently I did. Yeah. I, I think it was when we did the first podcast about it, which I think was before you even joined Decipher.

I think it was me, Pete and Lindsay or Zoe. Zoe. I was going to say yeah, I don't remember. Yeah. Went off to go rewatch the first one too. It was a long time ago. It might not have even been on video. I don't remember. But I do recall me trying to defend Michael Mann's filmmaking at some point. I don't know, but the the makes no sense whatsoever. But it's going to be so much fun to do this, especially with John, who, you know, I don't even know if he'll be able to explain like the technical things that happened in this movie because they're so they're so silly. But yeah, it'll be a lot of fun.

So that's coming at you in about a month. And we're also going to do Ferris Bueller. I think we should do that but before Black Hat because it's like it's springtime. Ferris Bueller is fun. It was just on. I was just watching it like 2 days ago with my wife. It just makes me smile. Yeah, I love that movie. It's so good. It's just a delight. Yeah. So. And it's always funny to see, you know, Cameron, like if you watch Succession, seeing him as like a however old he was in Ferris Bueller, like 19 or 20, just like, Oh my God. And he's on Rooster now, which I know that you watch. Oh yeah, I was just thinking.

I was like, I saw him in another show and yeah. Yeah, he's like a dirty old man on Rooster. He's just like, this is so weird. I don't, I don't like it. Yeah, it's so. Like his character on succession versus his character in Ferris Bueller's Like, very different. So different, Yeah. And Ferris Bueller's just like kind of a harmless, you know, hypochondriac, like, you know, anxiety ridden dude. And in succession, he's just like a clueless, you know?

Fumbling. Like, yes, tech bro, idiot, you know, the kind of people that run our economy right now. So yeah, yeah, there we go. But all right, that's probably enough depressing content for for one day. Yeah, we'll come. We'll be back next week with with better news. Yeah. Fingers crossed. We'll try and pick up the the optimism factor next week. Yeah. All right, Lynns, have a good weekend.

You too. See ya. Bye.