If last month’s CrowdStrike outage had you reaching for a business continuity plan but coming up empty handed, this post is for you. Business continuity planning is critically necessary for companies in every industry.
Recent events, like the CrowdStrike outage and far-reaching cyber attacks like the one on Change Healthcare, highlight our growing interconnectedness. Fewer vendors handle more of our business needs. Change Healthcare touches 1 in every 3 patient records in the U.S. And CrowdStrike is estimated to hold nearly 24% of market share in the endpoint protection business, serving 298 of the Fortune 500 and 8 of the top 10 financial firms.
Outages, attacks, and other disasters ripple further and faster than ever before, affecting real people’s lives and businesses’ bottom lines. Take the warning and consider how business continuity planning could make you ready for the next big event.
The Outage
Some are calling it the largest IT outage in history. On July 19, CrowdStrike released an update to its Falcon platform. To provide protection against cyberattacks, the platform must be tightly integrated with the Microsoft Windows OS. The flawed update to Falcon sensor version 7.11 caused the platform to crash, bringing the operating systems of millions of computers down with it.
While only 1% of worldwide systems were affected, they were the 1% running some of the world’s most critical and time-sensitive operations, including airlines, public transit, financial services, and healthcare systems.
Insurance provider Parametrix estimates that the outage caused $5.4 billion in direct losses for the Fortune 500. And Delta’s CEO says that his airline was hit to the tune of $500 million in losses. Insurance will likely cover only 10 to 15% of these costs.
What Went Wrong at CrowdStrike
This article is not really about rehashing what went wrong on the CrowdStrike side of the equation. Suffice it to say, that someone (or some team) had one of the worst workdays of their life in July. We can only speculate, but it seems that more testing should have been done before the release. And the update should not have been released to all customers at the same time.
Lawsuits are already being filed, with one alleging that the company “had instituted deficient controls in its procedure for updating Falcon and was not properly testing updates…”
For its part, Crowdstrike says that a bug in its Content Validator program allowed the flawed update to slip through.
Preparing for the Next Outage
There are more certainties in life than just “death and taxes.” Tech failures are going to continue happening as well. Every company should view the CrowdStrike outage as motivation to scrutinize their business continuity plans.
First it’s important to know the difference between a business continuity plan and a disaster recovery plan. The latter is mainly for your IT staff. A disaster recovery plan focuses on information and technology recovery after a disaster. A business continuity plan will involve staff from across your organization.
The basic steps of building your business continuity plan are:
- Business Impact Analysis
- Recovery
- Organization
- Training
Business impact analysis identifies the effects of many types of disruptions to your business’s functions and processes. These could include small, everyday interruptions like power outages or a large number of employees calling out sick. But the real trouble comes when critical systems go offline, data is ransomed, or worksites are unreachable due to natural disasters or other troubles.
An impact analysis should ask, if X no longer functions, what will the financial and operational impacts be? Is there a backup for X that will work as a failover? How long can the business function on the failover system for X? What are the costs for operating on the failover system?
For the recovery portion of planning, businesses need to identify steps to recover the critical business functions highlighted in the analysis. For recovery, you’ll need a team that is adept at running the failover system and another that is adept at recovering the primary system.
That’s where organization comes in. A solid continuity plan has a named continuity team that is on call to enact the plan for managing the disruption.
Finally, your identified team must be trained and tested. In the same way you pen test your security systems, your continuity team needs to complete exercises that cover the continuity plan and its strategies. And training doesn’t end with your named team. Much like a fire escape plan that all building occupants must be aware of on some level, a business continuity plan should be shared with your entire organization.
It’s Not a Matter of If…
While the CrowdStrike outage was the biggest event of its kind, it’s hardly going to be the last. Owning the fact that additional calamities (tech failures, cyberattacks, natural disasters) are definitely coming should inspire you to review and/or create a business continuity plan for your organization. Investing time and resources now can help prevent massive losses in the future.
If your organization needs to review its system failover plan, Asylas can help. Reach out at 615-622-4591 or email info@asylas.com. Or complete our contact form.
