Hyperscale Security

To document all of the weird things we run into in cloud security, I long thought someone should be the “rachelbythebay” for secure cloud transformation...

As part of the World Economic Forum annual event in Davos earlier this month, their Global Cybersecurity Outlook 2023, produced in collaboration with Accenture, was released. It's generally a reasonable report, but it got a lot more attention in the media because of one particular survey finding. As for instance reported here and here, their research found that found that 86% of business leaders and 91% of cyberleaders believe that global geopolitical instability is likely to lead to a far-reaching, catastrophic cyberevent in the next two years.

When we look at the data in the report how that breaks down for cyber leaders, we get this table.

Company Size Very Likely Moderately Likely Not Very Likely
< 250 50% 42% 8%
251 – 1,000 33% 67% 0%
1,001 – 10,000 47% 40% 13%
10,001 – 100,000 38% 52% 10%
100,001+ 73% 27% 0%

Even though the report does not provide details, it also states that 43% of organizational leaders think it is likely that in the next two years, a cyberattack will materially affect their own organization.

What Does a “Far-reaching, Catastrophic Cyberevent” Even Mean?

In the press conference at the release of the report, Edi Rama, prime minister of Albania gave an indication of what such a catastrophic event would look like.

“Let’s imagine an exponential multitude of viruses that mutate everyday exponentially while not threatening our body, but the bodies we live in, our organizations, our countries, our system, then, you know, it could be just apocalypse. It’s about viruses that can not only block our way of living, but can control it and deviate it.”

Albania has experienced significant cyberattacks attributed to Iran exposing intelligence and police data, but that sounds like incoherent nonsense to me.

But the man is a politician. What makes 91% of cyber leaders expect such a catastrophic event? Among those in organizations larger than 100,000 employees 73% consider it very likely and 27% moderately likely. That's curious when 57% of business and cybersecurity leaders don't think it is likely an attack would materially affect their own organization.

Something Doesn't Add Up

While across the board business leaders perception of their organization's ability to be cyber resilient dropped compared to last year, cyberleaders feeling confident their organizations are cyber resilient rose from 12% to 29%. An additional 54% perform common cyber-resiliency practices but recognize they have more work to do. Don't we always.

So, who then is this far-reaching, catastrophic event going to affect?

Surely, far-reaching and catastrophic means something bigger than Mirai, Solarwinds, Log4Shell, Rackspace, or Ransomware-as-a-Service.

If 91% of cyberleaders consider such an event likely, yet 83% consider themselves to have a baseline of cyber resiliency in place, this seems an irrational fear of such an event happening elsewhere.

Catastrophic for Whom?

I am a “cyber leader” (arguably 😉) in a cloud service provider with over 100,000 employees – supposedly the category that considers such an event very or moderately likely. We certainly threat model and try to mitigate against catastrophic scenarios. But despite the role the company plays for our customers, it is still very unlikely that a cyber event would grow into a far-reaching, catastrophic one for society.

That is not to say that a cyber event couldn't be far-reaching and catastrophic for a single or select customers, or the company itself.

So, I am going to be an outlier compared to the report. I consider a cyber event along the lines described by the Albanian prime minister highly unlikely. But I do consider it very likely that a cyber event proves catastrophic for a single large organization in the next two years. Our job, meanwhile, is to ensure we do what we can to prevent that being us.

cloud security posts without corporate approval @jaythvv@infosec.exchange

Relationships are funny things. You may be inseparable through college, but drift apart and find after a few years when you reconnect you have nothing to say to each other. Happy couples that love each other may go through major life changes that with no malicious intent can change the balance of the relationship.

Technology companies buy different technology from each other. Hardware companies need software, while software companies need hardware and other software to operate. Without major outside change, such strategic technology partnerships can last for decades.

Cloud Transformation as a Major Life Change

Cloud Transformation cascades in ways you often don't expect, including to your strategic partnerships. While moving from data center to the cloud, you may find you no longer buy that much hardware anymore. You may change your tooling to different vendors. You may no longer need network routers or firewall equipment to the same extent.

Unless your strategic partner was on that journey with you, you may find that you have drifted apart and no longer have much to say to each other. The partner may be going through their own transformation, but into a solution area that doesn't appeal to you. The balance of mutual spend might change, as your requirements and expectations may have.

Your partner may be utterly unprepared for that.

“Explore Ways to Help Solve For Your Initiatives”

We recently had an executive meeting with a long-time strategic partner. As usual, we had prepared an agenda in advance and shared our key priorities, approaches and challenges around managing the security of a large and fast growing Multicloud landscape.

The partner proceeded to present a vision that did not respond to that at all, but treated us as if we were still the software company of old, rather than the cloud company we are today. Suffice to say, the presentation did not resonate. It was barely coherent, given our priorities.

But we've been partners for years. There is a relationship. Nobody wants to start trouble. So you nod and politely close the conversation, and agree to a few high level action items.

Then the notes for the meeting arrive, with a reasonable summary of our priorities, followed by a proposal to explore ways to help solve for your initiatives through their product portfolio.

Our plans for 2023 are already established. Execution is already underway. We didn't expect you to solve for anything.

It's Not You, It's Me

How do you explain to a loved one you need to work through something yourself, even though they want to help? Maybe your little nephew offers to help finish your college work. Maybe you'd rather have another friend help you with more expertise in the area. Maybe you just want to figure it out yourself.

Thanks for the offer, but we just don't have enough shared experience for you to be helpful.

Cloud Transformation is hard enough within a single large organization. Change hits teams at different times and different business functions move at different speed. If not even everybody in your own organization fully grasps that you have changed, it is even harder for strategic partners to understand that.

If we are to keep a relationship alive after a life change, we need to adjust to the new reality and recognize that something has changed. We can still be friends, if we find a new balance. But if you insist on treating me the same way as before, this is not going to work out.

We need couple's therapy...

cloud security posts without corporate approval @jaythvv@infosec.exchange

Whether in security operations, infrastructure-, network- operations, SRE, etc., there is a certain attitude. We take the weight of the world on our shoulders keeping things running, and may even with some justification feel we care more about the systems we care for than the solution teams. We feel it in our bones.

It takes a particular personality. We work during vacation periods. We check our mail and chat channels out of regular hours. Not (just) because our boss asks us to, but because it is an integral part of what we do and who we are. You may say, that's not healthy. But we worry regardless, and feel more comfortable after checking that everything is good.

I was reminded of this when I saw this LinkedIn Post by laid-off Google SRE who despite being laid off, made sure that he handed over his oncall duties properly, in the process going over and above what could be expected. I could completely relate and left a comment of praise.

Only then did I see the comments. There were others like mine that recognized what he did. But also a lot of comments why do that for a company that has just laid you off? Let it burn...

I get the sentiment. I understand. I have been restructured before and it is not a pleasant experience.

But I am also quite sure in his position I would have done the same. We deal with bad stuff on a regular basis and work through it, and ensure proper hand-off. It's what we do.

A salute, therefore, to my brothers and sisters in ops – currently employed or not. o7

cloud security posts without corporate approval @jaythvv@infosec.exchange

It is a common refrain that security teams are under-resourced and under-staffed, and that they are not supported by the business. If that is truly the case, you're probably in the wrong place. But what often happens – and is something we at least have control over – is that we fail to communicate in a language that the business understands.

Talking To Ourselves

Many of us in #infosec come from technical background, and many get an introduction to our culture from hacker and cybersecurity conferences. The talks there are often highly technical, and even if more abstract, filled with jargon unique to the field. Vendors and analyst firms add an additional layer to this, that is hard to follow even if you are in the industry.

You are probably not going to get heard if you ask for funding to implement protections against the OWASP Top 10, deploy a WAF/IPS, cycle vulnerabilities with HIGH and CRITICAL CVE ratings, implement MFA and embark on a multi-year Zero Trust Architecture deployment.

Talk In Terms of Business Risk

Senior executives go from crisis to crisis and only what is most urgent will occupy their attention. They manage all kinds of business risks, of which security may not be on their radar at all.

But they do understand probability of risks and financial impact. Your organization almost certainly already has established scales for probability and financial impact thresholds that can be plotted on a 4x4 or 5x5 grid to articulate business risks. When security risks are communicated similar to how business risks are communicated, it is much easier for executives to compare where they sit relatively to other risks they need to manage.

Learn How to Prepare an Executive Deck

It seems only rarely anybody teaches you how to prepare a proper executive deck, and you tend to learn by participating in the process. As this process can easily lead to multiple editors changing language to the point of meaninglessness, the more you can get your deck into the expected shape, the less likely it is that people without domain knowledge will make changes before it is presented.

  • Everything you want to say must fit in 3 slides maximum
  • Make sure all that is most important to you is in the first slide — there is a realistic possibility you may not get beyond the first slide
  • Realize that many execs are high-speed information absorbers, so get to the point immediately. Make your point in 5-10 minutes

Prepare a Realistic Plan, including Budget and Number of Resources

The last thing you want is to succeed in getting attention and support, and have no answer when you are asked what you need to bring down the risk. That is your opportunity wasted. Make sure you have a budget and resource plan ready that shows what you are going to do, justifies the budget and articulates how this reduces business risks. Include clear milestones, dates and meaningful progress metrics that can be reported on.

Good communication typically comes from placing yourself into the position of others, and speak to them in language they understand. A good start there is to read any corporate strategy documents you can find, and align your message with its main messaging and how security supports that. You are bound to get a lot more traction that way.

cloud security posts without corporate approval @jaythvv@infosec.exchange

Recently, I sat through a presentation by a network-provider-turning-into-a-cybersecurity-vendor that showed a diagram of IaaS, PaaS and SaaS and the need to secure that full stack. I agree in principle.

Apparently, though, the big missing piece in that was Network Security as a Service.


There are many challenges I have been involved in working through in cloud security the past years. But that this was the big gap never occurred to me.

The Death of The Perimeter

The combination of the migration to public cloud and working from home/anywhere accelerated by the global pandemic has led to many people to declare the death of the network perimeter. Instead of resources being safely (...) protected behind firewalls in a trusted network, suddenly we have a multiplication of end points and network connections.

This is almost invariably seen as a big problem that adds to all the challenges we already face. An alphabet soup of solution categories are offered to get around this and everybody is talking about Zero Trust Architecture. I am not disputing these are good things, but they all require a lot of effort and money to implement and operationalize fully. It seems we don't even consider that there might be benefits to the perimeter's demise.

Free Micro Segmentation

And there certainly are. Each cloud account (subscription or project, depending on your platform) is an isolated virtual data center. Aside from wide-open default VPCs (which you should nuke or neutralize immediately), there is no network path at all between that cloud account, any other, or the outside world. Each has to be configured individually, or network segments opened. Voilà, free network micro segmentation and reduction of blast radius.

Out-of-band Administration

Whether you are using the cloud web admin console or, preferably, the cloud API and Infrastructure-as-Code, you are communicating with the cloud provider's API, not via a direct network connection to the landscape inside the cloud account. This takes place entirely out-of-band from a network perspective, and may as well be an air-gap.

If you really, positively, have to have SSH access to your landscape, you can easily configure the environment to only accept network connections from selected CIDR ranges.

Identity and Authorizations

With all due respect to everybody in networking, we absolutely appreciate everything you do to ensure that networks exist – but we less and less rely on network security to keep landscapes safe. That is, we essentially treat networks as inherently hostile. Network traffic should be encrypted end-to-end to a limited set of ports. Everything between network boundaries is an API communicating over 443.

The main security control is Identity and Access Management, or Cloud Infrastructure Entitlement Management (CIEM) if you like. Who (or what service) you are, how you authenticate yourself, and what authorizations you have is the main protection. What network path the request took is increasingly irrelevant while the network traffic is encrypted anyway.

Cloud Mindset

We shouldn't build controls for a new age based on the challenges and solutions of old. Building Maginot lines of lifted-and-shifted security solutions, patched up with new tools that try to recreate the safety of old is a fool's errand. It presupposes our “trusted” networks were safe in the first place.

We have to understand how the cloud work to properly protect it against the risks we face. This allows us to see new paths, and the power it provides as well. Policies applied at organizational level on all cloud accounts in the organization provide a level of enforced controls that are nearly impossible in the data center, without really costing anything. Central logging simplify how you monitor network flows. It's not all bad. Much is much, much easier.

When you hear us cloud hoodies go off on our mythical cloud-native mindset rants, this is the sort of stuff we mean.

cloud security posts without corporate approval @jaythvv@infosec.exchange

Cloud Security is constant crisis management. Not in terms of panic, everything is on fire, but as a constant mode of being. Should a fire breaks out at your house, I bet you'd be in panic mode, as I sure would be. But for the Fire Department it is just another Tuesday. (literally)

I was reminded of that when I saw two separate reports of vulnerabilities of the cloud come out just today:

Told You So Doesn't Stop Us From Having To Deal With The Impact

These are great write ups with technical details, and you should read them. But since they're already fixed, they are largely like a weather report for a different continent.

Hurricanes or tornados way over there may make the news but don't affect you as much as the showers or snow you may be dealing with back home[^1]. Scientists and activists have warned us correctly for a long time about climate change and made recommendations that for reasons outside their control weren't followed. First responders nevertheless have to manage the local disasters where they occur and do what they can with the resources they have available to them.

Security professionals similarly have offered good advice for decades. Also here often lack of resources, commitment or willingness to face consequences meant their recommendations weren't followed. Yet those in the trenches try to make systems secure and respond to alerts to the best of their ability, while security operations deals with the fall-out with the capabilities they have.

Managing Cloud Turbulence

Cloud transformation only adds to this sense of crisis. We rush head-first into cloud platforms and new cloud-native technologies like Kubernetes, little of it secure-by-default. We give more autonomy to developers, while we're still figuring out how the secure these landscapes. Changes to the way we work, develop and deploy code, and adjust organizationally to cloud transformation only add to the turbulence.

We only manage this by realizing we are in a state of constant crisis and will continuously have to make difficult choices. This is what crisis management is for. When you rarely have an incident, you may have good playbooks, but when an incident happens nobody knows what to do. When incidents are routine, you get a rhythm and comfort level with them. You manage.

That doesn't make me less concerned about these vulnerabilities – we rely on the cloud provider for the security of the cloud in our own threat modeling and processes. I am concerned about the impact of climate change on communities around the world, as well.

However, we all first have to deal with the crisis in front of us and prioritize according to the impact on our organization for the things we have control over and can mitigate against. It won't do to stand on the side lines and say “I told you so”. We have to save what we can. Welcome to constant crisis management.

[^1]: I am in the greater Santa Cruz area, recently hammered by massive rain storms. While our little community is OK, thankfully, many areas have seen major flood damage and the risk of landslides continues. Extreme weather events are obviously on my mind.

cloud security posts without corporate approval @jaythvv@infosec.exchange

How does anyone think this works?

Sponsored/InMail: Hi, my name is XYZ, Co-Founder and CEO at ABC, dedicated to solving the big problems in cybersecurity. Here's a link to my calendar so you can schedule a 30 minutes demo

Cybersecurity vendors, please stop doing this. I get a couple of these per day on LinkedIn or email, and the only thing it does is get you remembered for all the wrong reasons. Cold contacts like this literally scream that you have no idea what you're doing.

We all have a lot of work to do, and if you are in the cybersecurity industry, I assume you are in it to make the world more secure. Therefore, allow me to share some insight from the other end, and hopefully you can spend your energy better.

We Study the Market

Large organizations with a significant security organization follow what is going on. Different teams will track different segments through the industry media, analyst briefings and reports, and what we hear our peers are using and their experience. We attend conferences, watch webinars and listen to podcasts. We get regular dedicated presentations from established strategic partners. We have partner teams and investment arms that survey what the cybersecurity industry is doing.

You don't have to contact us – certainly not over LinkedIn sponsored direct message! – we will contact you. For all of our new cloud security tooling, we reached out to the vendor. Two of them we talked to for use cases we identified them for that they weren't even selling the product on. One of them, we contacted when they just came out of stealth, after an analyst had talked them up during a briefing and got us interested.

If we don't contact you, it's because you are not a realistic candidate. (Sorry)

Do Your Research

During my consulting days, before every job, I looked up their website and searched around to get a sense of what the organization was doing. If you give me in the first contact the idea you don't know what we do, how big we are, or what challenges we may face, you have just shown me you wouldn't even do that.

There are ~3,000 people in our organization involved in security in some form or fashion. We operate cloud services with critical workloads for paying customers. Established vendors have seen their products fail in our landscape. We have spoken on security and DevOps conferences about the challenges we've faced that are an easy search away.

Are you sure you would even want us as a customer? Are you ready?

I Get How This Would Be Good For You, How Is It Good For Me?

I have been in the tech industry since the 90s. I totally get why it would be a fantastic opportunity for you to have a customer this size and name. The marketing potential alone would be excellent, wouldn't it?

Cool. But what is the problem you are solving for us, though? What remaining gap do you cover? Why would we replace an established vendor for you?

At Least Give Me A Reason To Be Interested

The effort may not be entirely wasted. It's always possible that you are doing something very cool and innovative. It is useful to have a product sheet and a tech paper that allows me to place you in a particular segment. That still likely won't lead to a follow-up, but if it is interesting, I will remember you. If I hear others mention you, I will ask them about you.

There is at least one company, though, that keeps contacting me and never seems able to explain what they do. That is all I remember about that company. I talk to others about them and their sales approach. Don't be that company...

But My Product Is Different!

You may genuinely believe you have a solution I should know about and your startup is different from the pack. It may well be! So, tell the world, get onto conference talks and podcasts, and release technical papers. Get the attention of the analyst firms.

But realize that a company our size has 18-24 month procurement cycles with multiple layers of approval. You'd get fully scrutinized for multi-year viability, likelihood of being sold to a larger company, level of funding, and all that for the duration of a multi-year contract. You would have to prove why you're better than a preferred vendor that has more resources than you. You could run out of runway chasing a whale, when you could be building the business fishing for cod.

If you're good and get traction, and solve problems for your customers, we will notice you. We will follow your progress. And if we think you could fit in our plans, we will give you a call.

Just writing this, I already thought of enough material for a future follow-up: how not to screw up your first meeting. Stay tuned!

cloud security posts without corporate approval @jaythvv@infosec.exchange

About a week ago, I was in a quarterly security review meeting with one of the business units, including the relative business leader and BISO. All of us have been in the tech industry for decades. We discussed the progress of the last quarter in the middle of rapid cloud migration. As an aside, we agreed the three of us that cloud transformation is the hardest thing we've done in 30 years.

Over the last 4 years I was deeply involved in cloud security during a rapid cloud transformation that is ongoing (arguably, it never ends). As cloud transformation reaches into business functions and processes you never foresaw and poses challenges you never thought you would have, it is a journey filled with absurd and crazy adventures.

Hyperscale Security

I ran a company-internal blog to document what we were going through as a running commentary, simply because I felt some sort of record needed to be kept. Increasingly, I thought those struggling through their own cloud transformation could benefit from such experiences. I have always liked the 'war story' posts on http://rachelbythebay.com/w/ and thought I might do the same. I even got the domain name – hyperscalesecurity.com – but for months hadn't figured out what platform or how to host...

Jerry from Infosec.Exchange to the rescue! Long live the Fediverse.

cloud security posts without corporate approval @jaythvv@infosec.exchange