CrowdStrike Postmortem

Three resilience strategies

In the following section, we will explore various measures that can be implemented to prevent such problems from arising in the future. However, it’s important to keep in mind, that incorporating testing, governance, and controls can inevitably decelerate the deployment process. As a result, implementing these controls may reduce the risk of installing buggy software but increase the risk from a threat vector that your systems are not protected from. Therefore, the key here is to understand your systems and the risk profile they present.

• It is essential for a system accessible on the internet, with high customer engagement, to receive updates promptly to ensure full protection.
• For key operational infrastructure that is not internet facing, for example the application server that controls machinery on an assembly line, it is critical that the machinery always remains operational, so testing everything thoroughly is critical even if it doesn’t have the latest version.

In this balance of resilience, it is imperative to have a nuanced understanding of each of your systems and platforms and create appropriate policies that work for each, rather than implementing blanket policies across all systems. It is also a reminder not to become complacent in your testing.

1. Testing – don’t forget the basics

One of the most fundamental and important ways to prevent this kind of issue is to implement a testing framework. Ideally, the vendor (CrowdStrike) should take ownership of this process, as it’s unreasonable to expect customers to test multiple updates per day before they are rolled out. In any case, the CrowdStrike Falcon platform automatically pushes these updates to the endpoint, which underscores the need for thorough testing on the vendor’s part.

Bear in mind, other vendors update far less frequently, so setting and enforcing a policy where all updates are applied in non-production environments and functionally tested before deployment in production can prevent similar issues. This should be defined and enforced as part of an organisation’s change policy.

So how do you test multiple times per day?

To tackle the challenge of testing multiple updates per day, as CrowdStrike faced, an automated testing suite should be implemented as part of the CI/CD development pipeline. This suite can deploy each update to multiple platforms, perform a series of checks, and provide a report. All checks must pass before the update can be cleared for release.

In the case of the recent CrowdStrike incident, although some code-level checks were employed, they were too low-level and misconfigured, failing to detect the memory issue in the driver configuration. There was no further check at a higher level, meaning the update was never tested as compiled, installed code. This oversight highlights the need for more comprehensive and correctly configured testing processes.

2. Operational Controls – putting control back in your organisation’s hands

Further to testing, there are operational / governance controls that can be implemented that are specifically designed to slow down the deployment of updates to prevent buggy software from being deployed.

Typically, this takes the form of a policy such as n-1 where the version installed at a particular organisation is mandated through change policy as having to be one version behind the latest release. Depending on the organisation’s needs, this can be extended to two or even three versions behind, however, it’s important to check if the installed version is still supported by the vendor.

Another approach is d+10, where the new version is not installed on the organisation’s systems until 10 days after release. This ensures that someone else is the guinea pig to discover and iron out all the issues. Once again substitute whatever number of days is appropriate to the risk level of the system.

In both cases the customer needs to be in control of which versions are installed and must be able to opt to skip a version. This is something that can in most cases be negotiated with the vendor even when placed on an automatic update or release train platform.

3. Observability is essential – a proactive approach

Regardless of the policy setting and level of controls deployed to each of your systems and platforms, Observability is the essential ingredient to a resilient organisation.

Observability ensures smooth operations of your application and IT infrastructure, giving you complete visibility so you can detect problems before they escalate into full-blown incidents.

When done correctly, Observability collects appropriate health information from all your systems continuously giving you the ability to monitor specific components in real time or examine historical records from your systems. This proactive approach enables timely intervention and helps maintain system stability and performance.