Rich Mogull on the Cloudflare Outage, Resilience, and Single Points of Failure

It, it's that the, we have a greater dependency on them than we have before. There's, as you said, you know, I don't know, was it 5, is it 6-7 companies, whatever. Not the biggest list on the the face of the planet that we all come to rely on. So outages there are going to affect us. I mean, think about what's going on here. So with Cloudflare, there's what like kind of 3:00-ish companies that really dominate that area. We've got Cloudflare, we've got Akamai and then maybe Fastly in that maybe other ones I'm not aware of.

And now there's not any of those can affect a material percentage of the Internet with cloud, there's 3 hyper scale providers and one or two, you know, additional ones like the the fit in. So if Amazon, Google, Microsoft go out, that's going to affect a large swath of things and and the Oracle maybe is there, you know, maybe not. They're not where the others are these days or, or one of those others. So I've just listed 5 to 6 companies. If any one of those go out, a measurable percentage of the Internet becomes less reliable.

And there's kind of a way to avoid that because we're not running everything on premises. And if you ran all the data centre, you still need ACDM. So you know, I don't know monopolies. They're not monopolies, but it's close enough. Yeah, the one of the things that I always come back to when these things happen, and it's not a, it's not a one to one comparison, but the Windows monoculture paper that Dan Geer and Bruce Schneider and a few others wrote, you know, God, 20 years ago now or whatever it was, and that caused a lot of waves.

And they were not the first people to say, hey, monocultures are bad. It just riled up a bunch of people at Microsoft and got them mad. But monocultures are bad in, you know, biology in, you know, software in whatever it happens to be. If we all rely on the same one thing or very small handful of things, then you're going to get failures on grand scale. You're not going to get small failures where you're like, Oh, my local Internet provider, you know, like when we were growing up rich, like if you're, if your literal local town or regional dial up provider went down, a few 100 people got mad or maybe didn't even notice, you know, 'cause they weren't online all the time. But now when things, when something at the scale of Cloudflare or you know, AWS or Fastly or something like that has a big problem, as you said, everything relies on those now or some combination of those things. And there's not really a good way to rewind that, honestly. Well, it's not even a rewind.

The the, the economies of the systems we deal with require that you can't get the capabilities of a hyperscale cloud provider without having a hyperscale cloud provider, and the market can only support so many of those. The alternative is all the smaller data centers and such, but then there's no way they can keep up to the same degree with enterprise demands. So, you know, thinking about just the what's required for serverless auto scaling platform as a service and these are massive investments that give massive benefits. They do building on those architectures. We the Europe is seeing this right now as they're trying to develop, you know, sovereign clouds locally and you got OVH and you got stack it and they can't compete with the global providers, but the global providers themselves can't compete if they weren't global providers. So there's economies of scale

issues that just lead into this level of lack of diversification and I I don't see that going away. And now we do have a choice on which providers we pick and they are not all created equal. Now we do need to emphasize. Yeah, there's there are things that separate each of them and depending on your business requirements and what you're trying to achieve and maybe your regulatory requirements, geographic location, those sorts of things.

Yeah, that's probably going to play into which provider you're going to go with, Right. I mean, you mentioned some data sovereignty things. That's a big issue, especially in the EU. But one of the things that always happens when when an outage like this occurs is you start seeing people screaming into the Internet about some sort of government regulation or intervention or something like that. Exactly. You're, you're shaking your head for people that are only listening. That won't work in for a bunch of reasons, but the main one is you.

As you said, to be a hyperscale provider, you have to be a hyperscale provider. And there are only a handful of companies that can do that. IBM can't even compete with Google and Amazon. Like think about the size of IBM. They cannot. They're a minor player compared to AWS and and Google and Microsoft. Yeah. So you can't just create competition and be like, now you're a hyperscale cloud provider, go get them. It doesn't work that way. Well, and you know, even like like there's Europe so Dora to in this like from a regulatory standpoint, you know, we're pushing towards multi cloud, US regulators pushing towards multi cloud.

This is literally the conversation I had with a big financial right before this interview for our conversation. And there are some other regulators and I'm not the regulatory expert. I had one of my my Co workers, Troy Leach over here at cloud security lines. But he he's in that that realm and he's like, well, the regulators are starting to realize that the cost and complexity of multi cloud makes it not necessarily viable. And let's look at what we've got here. So the answer with a bunch of this stuff is we need to go multi cloud. Well, with Cloudflare, you can't like multi cloud for your CDN. Very few organizations can do that. But let's take a look at the Microsoft, AWS, Google thing. And all right, so if you went multi cloud there, the way you make the multi cloud work is you're probably sitting behind DNS and ACDN to route your traffic to the appropriate cloud provider.

So there's that point of failure. Or maybe it's like, well, we have to worry about, you know, managing our cloud and we have to manage multi cloud, which is really hard. So you're layer on another tool and I don't know, Wiz, Orca, Palo, whatever for a security folks. And then there's dependencies there and and they're themselves are multi cloud, but maybe. And so it's like turtles all the way down. One of my favorite, just like sum it up things like, I don't know, it's just turtles all Yeah, I don't know.

It was. All the way down, yeah. And I've had discussions and I, I'm positive you have too because this is what you do with organizations and Csos and CTOS who did the multi cloud thing. And then it didn't really work out all that well for them because of the complexity and the security issues and all of that. So they've gone back to just a single provider because it was simpler. And I don't know about maybe more cost efficient, but probably. Well, it's really, yeah. So a true multi cloud application, incredibly difficult to build those, incredibly difficult to manage those.

If it's truly one app that works on multiple providers. Containers get only get you so far because there's all the what we call the meta structure that you have to support everything from from DNS to security controls, monitoring and logging and and all of that. There's just like a lot of management playing things that aren't going to translate across all of those. So the fewer clouds where you have to run all of that stuff, it's more there's a manageability aspect to things because the none of the providers, the technologies are also fundamentally different.

Now realistically, larger orgs are always going to have multi cloud largely due to merger and acquisition activity or all their stuff. Or sometimes it's just sometimes it's bad business decisions, but sometimes it is legitimately, you know, they need to basically be able to support this because they just acquired somebody or something else. So it is really, these are very hard problems. Now, that said, many of these apps, if you build them, architect them even on a single provider correctly or will be more resilient than, you know, the creaky apps that you had in your data center. We tend to focus on the big public outages because a lot of things go out at once.

But you know, this from, I mean, we've been doing this forever. Like data centers go down, every connectivity goes down local. Like, you know, was it, was it South Korea just lost like a massive amount of stuff due to a fire in a data center, right? Yeah, it happens. And we only tend to notice it, you know, on a, on a big scale, when it is a big scale thing like AWS or, you know, Cloudflare or, or something like that, or, you know, if there's a Office 365 outage or, you know, whatever it happens to be. And everybody, myself included, loves to complain about nothing working anymore.

But the reality is we, we notice these things because they're still relatively rare. You know, things kind of work pretty well. You know, it's, it's a miracle in a lot of ways, given that, yes, the Internet was built to be a distributed system, but it wasn't built to be what it is. It was built to have about 8 nodes at universities. You know, like that's kind of it. If you go look at the original Internet map, that's it. Most of them are about 5 miles from where I'm sitting. And then there's a couple out by you.

That was it. Yeah. You know. Yeah, I mean, you're right. The and the other thing too is like, OK, when's the last time Netflix went down for you? But if it did, I would hear about it from my kids immediately it that's the early warning system. I think over 90% of what they were on is on AWS OK when that outage went out. Like we know how to architect for resiliency even within a single cloud. Now it is harder on on Microsoft on Azure because outages there tend to be global because of the way that it works. And, and Google's kind of got its own inner, you know, intricacies, all those things.

So it's not like we can't deal with these, but it's how much do you want to pay to deal with it? And some organizations have invested more and suffer fewer outages even when the providers have their outages because they put the investment in, they've architected for that. They've used chaos engineering, which I think is really powerful around this. And it's easy for me to say it because I don't have to build that resiliency. But even in my last employer, you know, my vendor job over at Firemon, like we just, we largely architected in smart ways that we haven't had.

Like even when Amazon goes out and that entire platform runs in AWS, we haven't had an outage, a significant outage of our platform. So, you know, you, you can deal with these. Could we have one? Well, yeah, absolutely. There's things that could take that down. But, you know, it's been almost, you know, 7-8, nine years without one kind of outage. From a security perspective, I think a lot of times we, you know, even for events like this that are not specifically security related, a lot of times security infrastructure can be affected by these. You know, it could be, you know, your MFA infrastructure, it could be, you know, I think some, some CAPTCHA things were down.

That's not really security, but it, it can get in the way of things that that need to happen for you to get into accounts. How do you see people actually dealing with that kind of stuff on the security side? How do they how do they sort of architect or build for those contingencies? Yeah. I mean, some of them can be architected around and some of them can't, and some of them don't matter. A lot of what we do in security, you know, I mean, for example, if something is down, then I probably don't need to protect. Yeah, but but silver tooling and stuff, the IT would usually affect the things that are more, you know, really a lot of what we do is it's a fail open versus fail closed is 1 layer of it. So I would, for example, I could get a greater security exposure if my WAFF is in a fail open for

certain values of WAFF or whatever you want to be. But most of the stuff we have will largely either fail close or the things that fail open is not like it's not the be all end all. I mean, this is where some of the defense in depth stuff comes in. So maybe I can focus more on detection. I've seen management planes go down. I think it was like Sentinel one had an outage. We know crowd strikes had some of those issues. I mean, everybody has it at a certain point where that control plane goes down, but it doesn't mean that 701 agents weren't working. It just meant things maybe weren't updating and you couldn't get into do your analytics and maybe some of your alerting.

You know, I mean, we, we do take that into account when designing security controls and it's a, but honestly, we usually don't have budget to do much about it. Let's be realistic for the larger outages because I'm not going to run 3 tools that do the same thing simultaneously that are not only going to cost more, but are going to come with performance hits and everything else. So there's aspects of layers and failover we do tend to have, particularly around some of our network security stuff. The endpoints will usually run independently unless the endpoint's the problem and takes everything down, which I don't know.

Hopefully, you know, we can get another five years before we have another one of those again. And and you know, so I'm kind of being nebulous because I see a lot of different things, but sometimes it's a hard problem, sometimes it's not. Just depends on which security controls that are and and you know how like active live critical is it in that moment that we can't handle an outreach for an hour? Yeah, I, I remember you doing some speaking and maybe some writing about like kind of the myth of resiliency, you know, in this, in this world over time. And one of the things that that made me think about is, and you just mentioned the budget part of it. There's only a handful of even very large enterprises that can build in any sort of like actual 1 to 1 redundancy for events like this.

You know, most companies even at large scale can't be like, Oh yeah, just switch over to our other massive cloud cloud provider and CDN and this and that for everybody else, you, you basically just kind of have to eat it for a while. Like there there's. Yeah, most of those large companies won't spend the money for that or don't have that, won't spend the money on the people with the knowledge to build that. Right. Yeah, that's a good point. And there's that group of people is relatively small too, I would think.

Yeah, yeah. So I mean, yeah, these are addressable problems. I don't like to use the word solved in a situation like this, but it's time, it's money, it's knowledge, and in some cases doesn't fucking matter. Sorry language, but. Please. I mean, so you know the CSA website went down when when Cloudflare went down, OK, Not the whole website. Like parts of what we did went down. So OK, come back later. Like your bank, banks used to not be open 24/7. Right.

Yeah. Well, there's transaction reconciliation and back end issues and all that. I don't mean to demean that, but when like, I mean honestly, if we didn't have social media, how many people would notice this stuff was down in the. 1st place No, that's exactly right. That's the way that most people probably were notified of this this morning and it was their main sort of interaction with it was some sort of outage, whether it's, you know, on Twitter or Instagram or whatever it happened to be. You were like, why isn't this loading? Why can't I see the dumb videos this morning and then you go and figure out it's a platform problem.

I mean, that's go. Outside people. I know that's literally what I said this morning, but I'm in here talking to you. The other thing I wanted to ask was we know that the security teams at these large providers, you know, Amazon, Fastly, Cloudflare, all of them, Akamai, very mature, very advanced, they think about this stuff all the time. They don't only think about it when bad things happen. They know how things break, why they're going to break, all that kind of stuff.

But there's always unintended or, you know, unanticipated consequences. And a lot of times it comes down to some small subtle change that was made, you know, probably with benign intent, and then there's a cascading failure. I think that was the case in the AWS outage from like a month ago, right? Yeah, there was the way that they updated and yes, it was, it was DNS that some of those updates were handled. They were it basically created a timing issue because one thing stalled out in a way that was.

Yeah, it was, it was fascinating when they revealed the details about this happened and they put in more things to deal with it. But, you know, even, there's just so many subtle ways and even with like robust mature chaos engineering, you, you're not going to figure out all the ways, you know, something can go out. And we we talk about brownouts versus blackouts in terms of, you know, did you, you know, did they like an AWS one? Did something get out of sync because one thing ended up running slower than something else and overwriting and everything else and locks?

And is even the smartest people in the world are just going to miss some of that stuff. There's no way you try to build resiliency in. So when something that happens in Amazon's case, the resiliency worked in that that was largely isolated to one region. And even the multi region impacts of that were relatively limited because they, they built to to deal with that as opposed to when front door went down on Azure and it took down, you know, the the entire Azure portal. And a lot of a lot of aspects related to that are when Azure's MFA went down and nobody could log in with the authenticator MFA issue.

So it's there's no way we can eliminate this complete. Yeah, I'm going to solve that for us. Oh my God, that was, I was kind of going to go down that road, but we don't have time for that though. Maybe the last thing I wanted to say was from an attacker perspective, if you're, you know, bad guy #3 in, you know, North Korea or China or whatever, and you're looking at ways to, you know, sow a little chaos in the US or the world at large. These are the kind of things that I'm. I'm certainly not giving this idea away. I'm not the first person to think about this. These are the kind of things that realistically you might be interested in causing rather than some, you know, sort of Stuxnet level virus put into the electrical grid. Oh.

But they already do like. You know, of course. Yeah, DDoS attacks from Iran against the financial providers, you know, and both Amazon and Microsoft has seen state sponsored level, you know, DDoS and and Dos attacks. We've got the the embedding in our power systems and everything else is to do exactly that. Yeah, the pre positioning, that's why they're there. Yeah, yeah, they're there. And then, you know, the US power grid, you, you don't have to take the whole grid down. You pull out enough of the nodes in the right places. And I'm not saying this is going to happen, but I'm not. It certainly has been strategized for decades going back to my Gardner days 20 plus years ago. We're talking about I hated the term digital Pearl Harbor, but that's what was like.

I hate that term, but I was I was on a project called that not my choice. Oh boy. On on that term, but stuff we were doing with the government to war game this stuff out. So yeah, I mean it, it's all known now. Some of the you can do subtle issues, but again, the they're not going to do these outages just to cause random chaos like the state sponsor. They got to time it. And even the the criminals, if they're going to do it, they're going to, you know, they're, they've got a profit motive, so they're going to time it.

Then you just get down to the anarchists. You just want to break things to break things. OK, that's fine. But you know, Russia has been doing this, I mean, particularly in Europe. And they time it to maximize, to get some kind of a result, either kinetic or financial or other. Yeah. I mean, you, you see the power grid in Ukraine go down in the middle of winter for, you know, at regular intervals and things like that. Yeah, it's yeah, not predictable, but you'd after the fact you're like, well, yeah, makes sense.

Yeah, where some of the information warfare, deception operations, campaigns, it's enough time to maximize effect. And I mean, that's just the nature of of that game. Yeah, that's their job. Yeah, they do. Yeah. Yeah. Alright, Rachel, thanks for your time, man. I appreciate you doing this on short notice. It's always good to catch up with you. Thanks a lot. Yeah, Yeah. I mean, do Dennis anytime and although you know, we just talked about work stuff, we can get in any of the funnel.

I know, but we'll, we'll do a longer one later. I know you're short on time, but I wanted to. I appreciate you jumping on to talk about this. Yep, no worries, I appreciate it. Take care. Thanks guys.