Breaking Down the F5 Breach

They don't say how long the attackers were in there. They don't attribute this attack to any, they don't even attribute it to a country or anything like that. It's just sort of the, the bare bones of it. And then, you know, it kind of goes into the the customer impact. But this is one of the more thorny and potentially really damaging ones that I think we've seen in a while. Yeah. And you know, just taking a step back, looking at F5 big IP, as soon as I saw that in the news, I was kind of like, OK, this is.

And I was just telling you, this is a product that has come up like, you know, a handful of times every year because it's such a popular type of appliance that threat actors have been targeting. And if you look at the, the platform, the product itself, it makes sense, right? I mean, if there's, it's on the edge of the network, it, there's kind of, if you get a foothold into that, it's easy access to then launch out lateral movement or some of these other vectors. But then also looking at F5 like the customer base I'd imagine is pretty valuable as well. For threat.

As if they want to target that. So it's, you know, just the reasons why, you know, the threat actors, especially nation state actors might be interested in this. I think is important to highlight here. The other thing too is, you know, you mentioned this kind of high level bare bones that was released yesterday. Since then, you know, there's been some news reports that have come out, I think from Bloomberg.

It was basically giving a little bit more information into, you know, what the scope and scale of this was, which, you know, it seems as though reports have linked the attack to China and then also that threat actors have actually been inside F5 network for 12 months, about a year. So just. That's a long time. That's a long time, yeah. If you really think about it, a year is of of access into your network.

Just think of all the things that happened and all the potentials. It's just hard to even know the true extent of this incident, you know? It really is. I, I can't really recall any publicly known incident where an attacker was inside of an A victim network for a year, 12 months. I mean, even going back to like the Aurora attacks that hit Google and a whole bunch of other companies back in like 2010 or 11, whatever year that was, they had long term access, but it wasn't anywhere close to that long.

You know, this is the kind of this sounds like, you know, whether these whether this was opportunistic. You know, the attackers had a bug that they were using against a bunch of potential targets and happened to get lucky and hit F5 with this. You know, with this, found a vulnerable instance of whatever and got in there and then, you know, may have sold that access to somebody or used it themselves, who knows. Or whether this was a targeted attack and they were specifically looking for, you know, looking to get into F5's networks and then see what they could get from there. Nobody knows.

But but the state back state backed actor targeted seems more likely. And there were a couple of sort of weird things or odd phrasing in the F5 statement. I'm just going to read like 1 portion of it from the What We Know section. And this came out on October 15th, which was the day everything became public. So it says we have confirmed that the threat actor exfiltrated files from our Big IP product development environment in engineering knowledge management platforms.

These files contain some of our Big IP source code and information about undisclosed vulnerabilities we were working on in Big IP. OK, next sentence. We have no knowledge of undisclosed critical or remote code vulnerabilities and we are not aware of active exploitation of any undisclosed F5 vulnerabilities. So those two statements seem to contradict each other. I'm not. The only way that I can sort of make them agree is if the attackers had access to this vulnerability database or whatever it was for X number of months, and now all of those vulnerabilities have become public. So everything that the attackers had access to is now public through other means.

You know, F5 has disclosed them and patched them or they've become public in other ways, but that that's not really the way that it's worded. So it's a little difficult to kind of parse all of that, but that that's the only way that I can parse it that makes any real sense to me. Yeah, that wording was weird. I think it's probably like legalese, you know, some sort of lawyer speak that probably got their their hands on that. To your point, though, I know they did also, they pushed out a security update and I guess it was like pretty big and it included some high severity flaws and then also some denial of service bugs that could be remotely exploited without authentication.

But like what, what worries me too about this is, you know, reading between the lines again. I mean, if threat actors got access to their source code, you know, looking at actual 0 days in the sense of, you know, the fact that threat actors now can find their own bugs and vulnerabilities that even F5 doesn't know about. And F5 may have disclosed all this. But it just makes me nervous that they can, they'll be able to study, you know, these things and really be able to get to kind of those root issues and those root weaknesses. And I don't think that's something that, I mean, they mentioned pen testing or whatever in the advisory, but like, I don't know if that was something that F5 has, you know, given me enough confidence in how they have talked about. They've addressed that.

The other thing too that I, that kind of came to mind was like, yes, they push out patches, but like if threat actors are able to have this level of access and able to really study, you know, this, you know, to this level like the source code and everything. Like, you know, I'd imagine if there was any sort of like incomplete patch or like, you know, something like just the ability for them to really skirt around that and, you know, have that level of knowledge, it just makes me nervous about what's going on behind the scenes.

Yeah, there's a, there's a bunch of different really concerning things there. If they, they got the source codes, there's all those problems that you mentioned. They did publish a couple of, they said that they brought in IO Active and NCC group to do, you know, kind of not the incident response work that you know, I guess they also brought in Mandiant and Crowdstrike. That's the kind of thing that they were doing. I think NCC and IO Active specifically were looking at the code repositories and things along those lines to see if there were any modifications.

I read the IO Active letter that said, you know, in the scope of the work that they looked at, they could, they didn't find any modifications to 2F5's code base. But that doesn't, as you said, Lance, it doesn't take into account the fact that the attackers can now just look at whatever they took. We don't know how much source code they took or what it was, but yeah, they can just kind of pour over that and get the lay of the land, see how see how everything works. You know, it's, it's yeah, it's really tough. Yeah, yeah, yeah.

The, and the other thing too I saw that was interesting was the, there were some reports that were linking this intrusion to a separate recent report by Google, Google Mandiant, which basically has some threat Intel parallels and talked about kind of Chinese espionage groups that were targeting, you know, tech companies to grab their source code data and analyze it for 0 days. And you know, I'm sure you've read that one, but it was the one that involved the malware, the Brickstorm malware. So that was a, you know, another kind of interesting piece of this as well.

Just kind of. Looking at the parallels there. I think, you know, if you, if you had to bet your mortgage on who the attackers were, I think you, I think we'd both bet on a China backed group. You know, I don't think that Google has a name for them in that report that you mentioned. It's one of the UNC groups, I believe one of the groups, not University of North Carolina, but yeah.

But the, the link to the Brickstorm malware is interesting too. They don't, there wasn't any mention in the Mandy or excuse me, the F5 reports from this week about any specific malware that they found in their environment or anything like that. Just judging by the way that these things have gone in the past, I would expect some sort of public post mortem report from either Mandy and or crowd strike on this. You know, it'll, it'll be limited because of the way these things work and Ndas and all of that.

But if there's some new malware strain or some use of unknown malware strain like Brickstorm, those are the kind of things that usually get publicly reported in these cases. And we haven't mentioned there's also SISA put out an emergency directive on Thursday or excuse me on Wednesday as well. You know, with the the usual language of that, you know, these things apply to all the federal civilian executive branch agencies. So basically if you've got an EF5-GO, look at it, see if it's vulnerable, if it is, you know, pull it off the network, all of that kind of stuff.

And there's, there's usually a 48 hour, 72 hour deadline for these things. I didn't, I don't recall exactly what it is for this one, but that gives you a good idea of, you know, obviously SISA knew about this long before it was public. That's the way that these things are coordinated. So I assume that they've been scrambling around on any other government networks as well, you know, the DoD and other networks trying to get those in shape. But at the same time, SISA is being systematically gutted.

So I don't really know how much manpower they have right now to to get to do this, to respond to something like this. You know, it's one thing to put out these EDS and, you know, public advisories, but it's, it's another thing like one of their functions is to go and help agencies respond to things like this. I don't know who's left to do that, you know? Yeah. And I don't even know who at other agencies themselves would be leading the charge on this. Great point.

If they're if they're there as well, I know you know, that's, that's definitely a good big picture thing to be thinking about right now. I I thought they saw that they had given them till like Halloween, which I thought was. Oh, OK, Long, yeah. Now that you're talking about this, it actually makes sense because they're probably trying to like, just like with the limited resources and manpower that they have in place now, it's like. Yeah, there's probably like 4 people that have to go around and do all this for every civilian agency.

Yeah, I mean, that's a whole other story. But it's, it's connected in that, you know, it's a limited pool of resources to do this stuff, you know? Right. But well, the you know, that brings to mind though the timeline here is interesting too. And the reason I bring that up is because I, I don't, I'd imagine there's been some behind the scenes like preparation from a government perspective, if the government is known about this. And it has because as you mentioned, they became, they F5 said they learned of this in August, August 8th or something. And but they only, you know, had the public SEC filing disclosed this past week.

But part of that is because the that you were mentioning the DOJ gave them an extension. So behind the scenes, you know, there has been some coordination that's not leaving federal agencies scrambling during a government shutdown. Yeah, it which if you go and read the SEC filing from F5, which is linked to in our story, you can see that DOJ, as you mentioned, I think it's September 12th is the date if I recall correctly, that DOJ said, yes, you can delay this basically out of the in the interest of national security because F5's products are everywhere, you know, including government agencies, as we mentioned, but they're in a ton of enterprises and cloud providers and everywhere else.

So it's one of those situations where it you know, there the longest running argument in the security community about when to disclose bugs and how to disclose them and all that kind of shit. But in these kind of cases, you look at it completely legitimate request on F fives part and to my mind completely legitimate answer from DOJ saying yes, in this case we should delay this so that everybody who can has a chance to go in, you know, make whatever changes they can in the interim. Right.

Yeah, that's that's a good point. And the other thing about timeline that stuck out to me, at least when I saw that report of that, you know, the threat actors had had access to the environment for a year essentially is, you know what, what was the tip off in August that eventually led to them discovering this? I mean, was it an external company or like did they figure it out themselves? And I don't know, like I feel like a year like there's granted it's Chinese threat actors, like they're pretty stealthy, I would say.

Professional had a lot. With like, you know, the malware and IO CS that Mandy and has disclosed over time. But like there's kind of, you know, there's there's always bread crumbs, like there's always clues that point to these types of intrusions. Like, yeah, yeah, it's 12 months. Like that just always is. That's just crazy to me. It is and it's, you know, my sort of educated guess, but not, you know, specifically to this is in a lot of cases what happens is there's a separate incident that seems completely unconnected.

And after the details of that become public, either an internal team or an external team goes and looks for indications of something similar, whether it was the exploitation of a specific bug or the use of specific malware like like Brickstorm. You know, somebody might have picked up that hunting guide and gone and looked and, you know, whether it was an internal team at F5 or somebody external was like, oh shit, this looks like we might have an issue here.

And then they sort of work back from there and you know, see where the attackers had access in it. But, you know, that's, that's one scenario that could be a bunch of others. But in a lot of cases, that's what you see with especially any kind of widespread vulnerability. You know, if it's some kind of like really widespread Windows bug or, you know, anything like that. And once one incident or intrusion becomes public, then everybody else starts looking around for that. They're like, oh God, yeah, you know, so I don't know. But that that's the most likely scenario I can think of. Yeah, I think you're right.

That's how a lot of those types of like discoveries are made. Yeah, I think. And the other thing too is like, you know, there's a couple of things that we still don't know at this point. Initial access is one of them. And so, you know, a lot of there's a couple of other pieces as well. But it it'll be, you know, interesting to see like how moving forward, if we see more of these, you know, facts or, you know. Findings come out through, you know, a Mandiant disclosure or something like that or how you know how F5 is going to kind of move forward in terms of revealing information about this. Yeah, I would assume they're going to continue to update what they can, you know in legally and and you know with the other constraints that they're going to have. But you know, in terms of the, the customer impacted, as you said, Lance, they, they released all these patches yesterday and you know, essentially said go

update, which is, you know, all the guidance there kind of is right now. And they're, oh, we should mention we, we didn't, it's my fault we didn't mention this. But another thing that F5 disclosed yesterday is that they rotated a bunch of their code signing certificates and keys as a result of this. They didn't say specifically what, but they did say that they listed a few big IP and other products and said if you haven't updated, you know, as a result of this key and certificate rotation, some of these updates might not take because, you know, if you haven't gotten there yet. So that sounds like another sort of artifact of this whole thing, you know, but that's, you know, it's just one other piece of it that's there's nothing customers can do about that, but just more kind of fallout.

Yeah, more fallout, yeah. I know, yeah. Like we need more of that. Yeah, no, we don't. But yeah, it it does seem like this is going to be one of those that we're going to be hearing bits as bits and pieces about for the next few days and weeks. I would imagine, you know, something else will come along and grab everybody's attention and then, you know, four weeks from now we'll see an update and be like, Oh yeah, yeah, I remember that. That was like 3 incidents ago. Looking at you like Avanti.

Sonicwall somebody. Yeah, I I saw her friend Matt Johansson yesterday, You know, somewhere, I don't know if it was Twitter or wherever was saying, you know, everybody has breached fatigue at this point. And it's just kind of one of those it professional hazards in this industry where you're like, OK, another thing, another thing, another thing. And it's hard to sometimes stop and take a look and be like, well, hold on. One of these things is not like the other. This is a big, this isn't just an attacker got in and stole some, you know, financial records from a manufacturer. This is a big thing.

So it we do sometimes, you know, your eyes glaze over at some point, you're like, OK, here we go again. But yeah, this is, I think this is a a little different than most of the other ones we usually see. Yeah, I think that's that is a really interesting topic. We should do another podcast on sometime though. It's just like the level of breach fatigue and like the issues with communication and the security industry about like, you know, how high, how much of A priority one thing is versus another. Because it is like, and you know, I think the the news honestly has has a big role and responsibility in this is there's just so many headlines over and over about, you know, there was this actively exploited vulnerability, There was this zero day.

But like what actually separates something like this type of incident from something from, you know, that we saw even earlier this week, we saw the Oracle extortion attacks, like continual fallout from that. So like looking at two different incidents, like what the impact is for customers, how long term is this going to be in terms of the effect? You know, all these questions, I think, you know, it's there's just a lot of a lot of different pieces that go into it.

Yeah. I mean, we, we both know people that do incident response and all of that kind of stuff. And those people do often have that kind of like 1000 yard stare where they've seen so many things. There's, you know, they're immune to those the same way that, you know, like EMTs have a certain look where they're like they've seen everything at some point. But if you're a customer or, you know, an internal security team, you have to treat every one of these things as its own, you know, discreet animal and be like, OK, let's go and look at this on its own. And you can't, you can't afford to let that fatigue really pile up on you, because then you can't really do the job that you're supposed to be doing, you know? Yeah, that's true.

Makes like really difficult it. Does yeah. And also even looking at the the advisory to like just the the F5 advisory, for instance, like, you know, using these same types of languages like long term persistent access, like I know this is never going to happen. I know this is just me being optimistic, but like if you're telling your customers that it was 12 months, just come out and say, say it was 12 months. Like we're going to find that anyway. Yeah. Yeah. It's it's just like being able to better communicate so that we can really fully understand the impact of some of these things.

Yeah, there are those coded phrases. It's, you know, long term persistent is something that you would look at. OK, That's, you know, people generally won't put long term in a statement unless they absolutely have to. You know, they'll usually say limited or, you know, whatever it is, they'll they'll use some parameters to make it sound squishier than it than it really is. And, you know, obviously this doesn't sound squishy at all. It sounds terrible.

So when I first saw that, I was like, yeah, this is, you know, that's one of the phrases you kind of stop on. So, yeah, it's again, I, I think we're, we have certainly not seen the the end of the disclosures for this so. No, definitely not. Yeah. All right. Well, we'll be definitely updating the piece that we wrote on this, and I have a feeling we'll be mentioning this on future podcasts as well. So yeah, stay tuned. Yeah.

All right. All right. Thanks, Lynns. Good to see you. Thanks. Bye. Bye.