On 19 July 2024 at 04:09 UTC, CrowdStrike released a content update for its Falcon sensor on Windows. Machines that received it crashed, and many could not restart normally. CrowdStrike reverted the update at 05:27 UTC, 78 minutes later, but by then the faulty file was already on every affected machine. Microsoft estimated that about 8.5 million Windows devices were hit, less than one percent of all Windows machines. Airlines, hospitals, banks, broadcasters and some emergency call centers lost systems that morning, and some organizations needed days to recover.
CrowdStrike said the same day that this was not a cyberattack. For continuity planners it was something harder to plan around: a trusted tool with deep access to every Windows machine failed everywhere at once, and the fix needed hands on keyboards, one device at a time. Below is the public record, then the lessons worth writing into your plan.
What happened
| Date and time (UTC) | Event |
|---|---|
| February 2024 | New sensor capability introduced, with predefined fields for Rapid Response Content |
| 5 March 2024 | First Channel File 291 content released after a stress test; three more updates in April work as expected |
| 19 July, 04:09 | New Rapid Response Content reaches Windows hosts. The sensor expects 20 input fields and the update supplies 21; an out-of-bounds memory read crashes the system |
| 19 July, 05:27 | Update reverted. Hosts on sensor 7.11 or later that were online between 04:09 and 05:27 may be affected; Mac and Linux hosts are not |
| 20 July | Microsoft estimates about 8.5 million affected devices and puts hundreds of engineers on customer recovery |
| Following days | Microsoft releases a tool that builds bootable USB media to remove the faulty file |
| 29 July | CrowdStrike reports about 99 percent of Windows sensors online, against the pre-update count |
| 6 August | CrowdStrike publishes its root cause analysis |
The root cause analysis says the bug could not be exploited by an attacker, a finding a third-party review confirmed. It lists the fixes: validation of the number of input fields, more checks in the content validator, added bounds checking in the sensor, a deployment ring process so new content passes successive rings before wide release, and new options that let customers control how Rapid Response Content reaches their machines.
Who was hit, and how
- Airlines. Flights were grounded on the morning of 19 July. The US Department of Transportation said most carriers recovered within a couple of days. Delta took longer and canceled about 7,000 flights, and on 23 July the department opened an investigation into why its recovery lagged; news reports pointed to trouble with its crew tracking system. Delta later put the cost of the disruption at about $550 million.
- Hospitals. Many US hospitals canceled nonurgent surgeries and appointments for the day while keeping emergency care open; Mass General Brigham in Boston was among them. In England, NHS England said a problem with EMIS, an appointment and patient record system, was disrupting most GP practices, which fell back on paper records and handwritten prescriptions.
- Emergency services. Some US 911 call centers reported problems in the early hours and switched to downtime procedures.
- Broadcasters and banks. Sky News in the UK went off air for part of the morning, and banks and retailers in several countries reported disruption.
Why some recoveries took days
The repair itself was short. On an affected machine, an administrator boots into Safe Mode or the Windows Recovery Environment, deletes the faulty channel file (its name begins C-00000291, in the CrowdStrike drivers folder), and restarts. Three things turned minutes per machine into days per organization.
- Physical access. A machine that will not boot cannot be fixed by a remote management tool. Someone has to sit in front of it, or talk its user through recovery mode by phone. That meant laptops in employees' homes, kiosks in stores, screens in airport halls and servers in data centers. Cloud servers needed their disks detached and repaired from another instance, which our piece on cloud region outages covers.
- Disk encryption. Where BitLocker protected the drive, the recovery-environment route needed that machine's recovery key. Microsoft's tool offered a Safe Mode route that avoids the key on devices using TPM-only protectors; otherwise every encrypted machine needed its own key. A key store running on Windows servers that crashed too has to be recovered first.
- Order and headcount. Directory servers, the key store and management consoles have to come back before most other repairs can move quickly, and IT teams are sized for a normal Friday, not for touching every device in the company.
Lesson 1: concentration risk sits inside your trusted tools
Concentration analysis usually looks at suppliers: one cloud region, one payment processor, one port. July 2024 showed it belongs inside the IT estate too. A security agent, remote management tool, device management platform or patching service touches every endpoint by design, so when it misbehaves, it misbehaves in the primary site and the recovery site alike.
Concentration also reaches you through other people's tools. Plenty of firms with no CrowdStrike license lost the day because their airline, hospital or service provider was down. Our guide to mapping third-party and supply chain dependencies shows how to find those fourth-party links.
What to do about it:
- List every tool with privileged, fleet-wide reach, and how each vendor pushes updates.
- Ask each vendor how updates are staged, whether you can hold back or delay updates for critical machines, and how fast a bad update can be withdrawn. The controls CrowdStrike added afterward, staged rings and customer control over content updates, are a fair baseline to ask any vendor about.
- Keep a small pool of devices for the crisis team that does not share the main stack, such as a different operating system or a separately managed configuration with its own protection. On 19 July, Mac and Linux hosts were not affected.
Lesson 2: plan manual recovery at scale
Write "every Windows device needs hands-on repair" into the plan as a scenario of its own:
- Keep BitLocker recovery keys retrievable when the main directory is down: an export held offline under the same controls as other break-glass secrets, plus a tested way to read keys to users over the phone after verifying who they are.
- Prepare bootable recovery media in advance. Set a repair order: identity and directory servers, the key store, management tooling, then front-line devices that earn revenue or protect life, then everything else.
- Count hands. Work out how many devices one technician can repair in an hour, including identity checks for key requests. Measure it in a functional test; our guide to business continuity plan testing shows how to score the result against a target. That number tells you whether your endpoint recovery time is a plan or a hope.
- Script the walk-through for staff at home, and give the help desk a callback queue that does not run on the crashed laptops.
Lesson 3: workarounds that do not need the endpoint
The NHS response is worth copying: GP practices fell back on paper records and handwritten prescriptions, measures NHS England said it already had in place. A workaround only helps if it runs without the failed devices. Paper order forms, printed contact lists, a cash-only or phone-order mode, a manual dispatch sheet: each should be printed, stored where staff can find it, and practiced once a year. If the workaround lives in a file on the laptop that will not boot, it is not a workaround.
Lesson 4: communicate when your own screens are the outage
Where staff were told what to do by desktop alert or corporate email, the channel and the incident were the same thing. Instructions need a route that runs on personal phones. Front-line staff facing queues without systems need a short script: what happened in one sentence, what you are doing, what the customer can do now, and when the next update comes. The Department of Transportation also said it would look at complaints about long waits for help during Delta's disruption, a reminder that how customers are treated over a long recovery becomes part of the story.
Lesson 5: it is a recovery event and a continuity event
IT disaster recovery brings the devices back. The business continuity plan keeps the business running while they are down and decides what customers and regulators hear. An outage like this needs both plans on the same timeline with a clear handoff, and our explainer on business continuity plan vs disaster recovery plan sets out who owns which part.
What to put in your plan now
- A scenario where every endpoint running your most widely deployed agent fails at once, with a named owner.
- The list of fleet-wide tools and the update controls each vendor offers.
- An offline, access-controlled route to disk encryption recovery keys, and a measured repair rate per technician.
- Paper or offline workarounds for every activity with an RTO under 24 hours.
- A staff channel that runs on personal phones, tested once a year alongside a timed repair of real devices.
Sources
- CrowdStrike, Remediation and Guidance Hub: Channel File 291 Incident, and Executive Summary: Root Cause Analysis, Channel File 291, 6 August 2024.
- Microsoft, David Weston, Helping our customers through the CrowdStrike outage, 20 July 2024, and Microsoft support article KB5042429 on the recovery tool.
- NBC News, 8 August 2024, on Delta's cancellations and reported cost; ABC News and Spectrum News, 23 July 2024, on the Department of Transportation investigation.
- NHS England's statement of 19 July 2024 as reported by Digital Health; Healthcare Financial Management Association reporting on US hospital disruption, July 2024.
- PA Media reporting on Sky News going off air, 19 July 2024.
Frequently asked questions
Was the CrowdStrike outage a cyberattack?
No. CrowdStrike said on the day that it was not a security incident or cyberattack, and its root cause analysis traced the crash to a faulty content update: the sensor expected 20 input fields and received 21. Its own analysis and a third-party review found the bug could not be exploited by an attacker.
How many devices did the CrowdStrike outage affect?
Microsoft estimated about 8.5 million Windows devices, less than one percent of Windows machines worldwide. The damage was out of proportion to that share because many affected machines sat in airlines, hospitals, banks and other organizations running critical services.
Why did some organizations recover in hours and others in days?
The repair was simple but manual. Organizations with recovery keys at hand, staff near their devices and a clear repair order moved quickly; those with encrypted laptops spread across homes or too few technicians took far longer.
Should we stop relying on a single security vendor?
Not necessarily. Endpoint protection is not optional, and two agents on every machine bring risks of their own. Control how vendor updates reach critical machines, keep a small, differently configured pool of devices for the crisis team, and plan the manual recovery as if it will happen again.
