A faulty component in the latest CrowdStrike Falcon update is crashing Windows systems, impacting various organizations and services across the world, including airports, TV stations, and hospitals.
The glitch is affecting Windows workstations and servers, with users reporting massive outages that took offline entire companies and fleets of hundreds of thousands of computers.
According to some reports, emergency services in the U.S. and Canada have also been impacted.
Workaround for CrowdStrike glitched update
For the past few hours, users have been complaining about Windows hosts being stuck in a boot loop or showing the Blue Screen of Death (BSOD) after installing the latest update for CrowdStrike Falcon Sensor.
The security vendor acknowledged the issue and published a technical alert explaining that its engineers “identified a content deployment related to this issue and reverted those changes.”
“Symptoms include hosts experiencing a bugcheckblue screen error related to the Falcon Sensor,” CrowdStrike says in the tech alert.
The company revealed that the culprit is a Channel File, which contains data for the sensor (e.g. Instructions). Since it is just a component of the update for the sensor, this type of file can be addressed individually without removing the Falcon Sensor update.
For those already affected, CrowdStrike provides the following workaround steps:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:WindowsSystem32driversCrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
George Kurtz, the President and CEO of CrowdStrike announced a few minutes ago that the company “is actively working with customers” and confirmed that the problems are caused “by a defect found in a single content update for Windows hosts.”
“We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers” – George Kurtz
CrowdStrike’s CEO says that a fix is available and advises customers to access the support portal for the latest updates.
source: George Kurtz
In an updated statement, CrowdStrike says that “the problematic channel file [C-00000291*.sys” with timestamp of 0409 UTC] has been reverted” and the good version of it is C-00000291*.sys with timestamp of 0527 UTC or later.
The company also provides two options to address the issue in cloud and virtual environments, one variant being to roll back to a snapshot before 04:09 UTC. The second option is the following seven-step procedure:
- Detach the operating system disk volume from the impacted virtual server
- Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
- Attach/mount the volume to to a new virtual server
- Navigate to the %WINDIR%System32driversCrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Detach the volume from the new virtual server
- Reattach the fixed volume to the impacted virtual server
Outage hits airlines and hospitals worldwide
By the time of the correction, though, many large organizations across multiple verticals had already been affected.
Some reports say that CrowdStrike’s update impacted some 911 emergency service agencies in the state of New York (EMS, police, fire department), Alaska, and Arizona, as well as 911 services in parts of Canada.
A 911 telecommunicator in Illinois said that they were “working off of paper until things come back.”
There also reports that the health hotline in Catalonia, Spain, is impacted and authorities are asking citizens not to call 061 unless there is an emergency.
Dutch broadcasting organization NOS said that the glitch created disruptions at Schiphol Airport and “forced several flights to be grounded” (operated by KLM and Transavia).
Melbourne Airport said that it was experiencing “a global technology issue which is impacting check-in procedures for some airlines.” The most affected are passengers departing internationally via Jetstar and Scoot airlines.
A few hours ago, in the latest update, the Zurich Airport says that “flights with destination Zurich that are already in the air are still allowed to land,” no aircrafts “are currently taking off for Zurich Airport,” and there are no departures to the U.S.
Furthermore, there are delays and cancellations and passengers of individual airlines must be checked in manually.
Other airports affected are in Berlin, Barcelona, Brisbane, Edinburgh, Amsterdam, and London.
In the U.S., the Federal Aviation Administration received requests to assist multiple airlines (American Airlines, United, Delta) with ground stops until “a technical issue impacting IT systems” is resolved.
On JFK and LaGuardia airports in the U.S., flights have been grounded due to outages from the CrowdStrike update, leaving passengers stranded.
Some hospitals in the Netherlands – Scheper in Emmen, Slingeland Hospital in Achterhoek, and emergency posts in Hoogeveen and Stadskanaal were also impacted.
In Barcelona, the Terrassa University Hospital and the Catalan Oncology Institute experienced issues earlier today due to the CrowdStrike issue but have started to return to normal activity.
In the U.S., Bellevue hospital in New York and NYU Langone Hospital are also impacted.
On Friday morning, multiple television stations and news outlets, such as Sky News and ABC suffered disruptions as computers crashed.
A large number of users started to spill their frustration in Reddit comments about tens and even hundred of thousands of computers crashing after CrowdStrike’s update and the impact on their companies:
Malaysia here, 70% of our laptops are down and stuck in boot, HQ from Japan ordered a company wide shutdown
210K BSODS all at 10:57 PST….and it keeps going up…this is bad….
Workstations and servers here in Aus… fleet of 50k+ – someone is going to have fun.
Failing here is Australia too. Our entire company is offline
Same here in OZ. Entire company is down.
Half the company down. Somehow it has hit our AWS servers also. Major service downtime for our customers
Entire org and trading entities down here. Half of IT are locked out.
Seeing major issues here in NZ at the moment, company wide outage impacting servers and workstations.
Supporting Philippines and China Locations. All experiencing the same as well
Despite a fix being deployed and CrowdStrike providing a workaround for Windows hosts already crashing, companies will feel the effects from the issue for a while.
Admins are going to have a long weekend, especially with computer fleets of tens or hundreds of thousands of computers, employees working remotely, off-premise data centers, or cloud environments where booting in safe mode is not an option.
Update [July 19, 09:59 ET]: Article edited to include mitigation details for cloud and virtual environments.
Source: www.bleepingcomputer.com