CrowdStrike: the illusion and risk of “too big to fail”

The CrowdStrike Falcon Sensor is an advanced security software designed to protect computer systems from potential threats.

On July 19, a Falcon Sensor update from CrowdStrike caused severe issues for Windows users worldwide. It introduced a defective file (csagent.sys) that led to system crashes and BSOD (blue screen of death) on millions of Windows devices. The problem affected various sectors, including airports, banks, hospitals, and businesses, causing significant disruptions in daily operations.

The incident with the Falcon Sensor update from CrowdStrike highlighted the importance of careful and secure software update management. This episode revealed several critical points that companies must consider to protect their operations.

We usually expect every software update to be thoroughly tested in a test environment before being widely distributed. Thorough testing helps identify issues and prevents disruptions in business operations. A structured update management process, with strict controls, reduces the risk of introducing vulnerabilities or causing malfunctions. The CrowdStrike incident showed that proper testing could have identified and fixed the defective file (csagent.sys) issue before release.

A poorly managed update caused a global system crash, demonstrating the need for rigorous and reliable testing processes.

The Business Continuity

Every organization must rely on its own effective and regularly tested business continuity plan, which includes crisis scenarios and rapid response strategies. In this specific case, organizations discovered the ineffectiveness of their business continuity plans because they did not prepare for such an issue.
Can we anticipate such a threat? Maybe yes, but it was probably at the bottom of the priority list. Perhaps it could, but it was likely at the bottom of the priority list.

The need for technical staff

Another critical point is the need for on-site technical staff to quickly resolve a relatively simple problem. Let’s take airports as a concrete example. To restore operations and allow passengers and crew to board, technicians needed to delete the defective file immediately. If the affected devices also had disks encrypted with BitLocker (which we should hope they did), technicians needed to retrieve the unlock key for each device (where is it stored?) to intervene. With 8.5 million PCs affected, and assuming each technician could unlock about 100 devices, with an average processing time of 10 minutes per computer, we needed about 85,000 technicians immediately. Deploying such a large number of technicians quickly presents a huge challenge and highlights the lack of preparation and the need for well-structured continuity plans.

The complexity of the IT supply chain

Windows is the most widely used operating system in the world. It develops much of its software internally, and to integrate and enhance features and security, many companies also rely on third-party solutions.
This approach, while offering flexibility and access to advanced technologies, introduces additional risks:

• Long and complex supply chain: Managing a supply chain with many intermediaries increases costs and complicates effective monitoring. Lack of transparency and control over suppliers are significant issues.

• Dependence on third parties: Relying on external software and services exposes companies to vulnerabilities outside their direct control.

• Delays in responses: Third-party vulnerabilities require longer response times, as companies must coordinate with various suppliers.

In contrast, IT service providers who develop and manage all their solutions internally—operating in a more controlled environment—enjoy greater control over their systems.
This approach reduces the risks associated with third-party updates and ensures greater overall stability and security.
These systems inherently have fewer points of vulnerability and are less susceptible to third-party systemic failures.
For companies, especially those managing critical operations, relying on providers that offer end-to-end internal solutions could ensure greater reliability and operational security.

Blind trust in Big Tech.
Are they really “too big to fail”?

We often consider the technology from big companies as infallible, without the need for external checks, leading to a lack of third-party oversight. Big tech companies claim to self-regulate and ask us to trust their assertions; this incident shows that trust alone is not always enough. We need greater external control and independent oversight to ensure the safety and reliability of technological solutions.

We often regard technology from big tech companies as a doctrine, almost forcing users and businesses to trust blindly. This trust means accepting Big Tech solutions without question, based on the belief that their size and resources guarantee quality and reliability.