Businesses across the world have been hit by widespread disruptions to their Windows workstations stemming from a faulty update pushed out by cybersecurity company CrowdStrike.
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts," the company's CEO George Kurtz said in a statement. "Mac and Linux hosts are not impacted. This is not a security incident or cyber attack."
The company, which acknowledged "reports of [Blue Screens of Death] on Windows hosts," further said it has identified the issue and a fix has been deployed for its Falcon Sensor product, urging customers to refer to the support portal for the latest updates.
For systems that have been already impacted by the problem, the mitigation instructions are listed below -
- Boot Windows in Safe Mode or Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Find the file named "C-00000291*.sys" and delete it
- Restart the computer or server normally
It's worth noting that the outage has also impacted Google Cloud Compute Engine, causing Windows virtual machines using CrowdStrike's csagent.sys to crash and go into an unexpected reboot state.
"After having automatically received a defective patch from CrowdStrike, Windows VMs crash and will not be able to reboot," it said. "Windows VMs that are currently up and running should no longer be impacted."
Microsoft Azure has also posted a similar update, stating it "received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines" and that "several reboots (as many as 15 have been reported) may be required."
Amazon Web Services (AWS), for its part, said it has taken steps to mitigate the issue for as many Windows instances, Windows Workspaces, and Appstream Applications as possible, recommending customers still affected by the issue to "take action to restore connectivity."
Security researcher Kevin Beaumont said "I have obtained the CrowdStrike driver they pushed via auto update. I don't know how it happened, but the file isn't a validly formatted driver and causes Windows to crash every time."
"CrowdStrike is the top tier EDR product, and is on everything from point-of-sale to ATMs etc. – this will be the biggest 'cyber' incident worldwide ever in terms of impact, most likely."
Airlines, financial institutions, food and retail chains, hospitals, hotels, news organizations, railway networks, and telecom firms are among the many businesses affected. Shares of CrowdStrike have tanked 15% in U.S. premarket trading.
The Texas-based firm, which serves over 530 companies in the Fortune 1,000, develops endpoint detection and response (EDR) software that are given entrenched, privileged access to the operating system's kernel to flag and block security threats. However, this access also gives them wide-ranging powers to disrupt the very systems they are trying to secure.
"The current event appears – even in July – that it will be one of the most significant cyber issues of 2024," Omer Grossman, Chief Information Officer (CIO) at CyberArk, said in a statement shared with The Hacker News. "The damage to business processes at the global level is dramatic. The glitch is due to a software update of CrowdStrike's EDR product."
"This is a product that runs with high privileges that protects endpoints. A malfunction in this can, as we are seeing in the current incident, cause the operating system to crash."
The recovery is expected to take days as the problem needs to be solved manually, endpoint by endpoint, by starting them in Safe Mode and removing the buggy driver, Grossman pointed out, adding the root cause behind the malfunction will be of the "utmost interest."
Jake Moore, global security advisor at Slovakian cybersecurity company ESET, told The Hacker News that the incident serves to highlight the need for implementing multiple "fail safes" in place and diversifying IT infrastructure.
"Upgrades and maintenance to systems and networks can unintentionally include small errors, which can have wide-reaching consequences as experienced today by CrowdStrike's customers," Moore said.
"Another aspect of this incident relates to 'diversity' in the use of large-scale IT infrastructure. This applies to critical systems like operating systems (OSes), cybersecurity products, and other globally deployed (scaled) applications. Where diversity is low, a single technical incident, not to mention a security issue, can lead to global-scale outages with subsequent knock-on effects."
The development comes as Microsoft is recovering from a separate outage of its own that caused issues with Microsoft 365 apps and services, including Defender, Intune, OneNote, OneDrive for Business, SharePoint Online, Windows 365, Viva Engage, and Purview.
"A configuration change in a portion of our Azure backend workloads, caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections," the tech giant said.
Omkhar Arasaratnam, general manager of OpenSSF, said the Microsoft-CrowdStrike outages underscore the fragility of monocultural supply chains and emphasized the import ance of diversity in technology stacks for greater resilience and security.
"Monocultural supply chains (single operating system, single EDR) are inherently fragile and susceptible to systemic faults – as we've seen," Arasaratnam pointed out. "Good system engineering tells us that changes in these systems should be rolled out gradually, observing the impact in small tranches vs. all at once. More diverse ecosystems can tolerate rapid change as they're resilient to systemic issues."
Falcon Fallout Unleashes Phishing Attacks
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has warned of malicious actors attempting to capitalize on the worldwide IT disruption caused by a flawed software update to the CrowdStrike Falcon EDR platform that has crippled countless Microsoft Windows computers.
"Threat actors continue to use the widespread IT outage for phishing and other malicious activity," the agency said, urging "organizations to ensure they have robust cybersecurity measures to protect their users, assets, and data against this activity."
This includes setting up scam domains and phishing pages – crowdstrikebluescreen[.]com, crowdstrike-bsod[.]com, and crowdstrike0day[.]com – purporting to offer remediation and recovery scripts to address the BSoD issue in exchange for a cryptocurrency payment.
Other kinds of activity observed range from impersonating CrowdStrike staff in phone calls to posing as independent researchers and claiming to have evidence the technical issue is linked to a cyber attack.
CrowdStrike, which has since apologized for the havoc caused by the botched update, said it "understands the gravity and impact of the situation" and that it's aware "adversaries and bad actors will try to exploit events like this."
The company has also shared additional technical details that led to Windows systems experiencing what's called a boot loop following the configuration update, noting it's currently performing a root cause analysis to determine "how this logic flaw occurred."
"Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform," it said. "This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems. The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC."
In the aftermath of the disastrous glitch, Microsoft said it's "working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online."
