Banks, airlines, hospitals and government agencies around the world were thrown into chaos. CrowdStrike released information to repair affected systems, but experts said it would take time to bring them back online because the faulty code would have to be removed manually.
“Maybe because of the vetting or sandboxing that we do when we look at code, this file wasn’t included or slipped through,” said Steve Cobb, chief security officer at Security Scorecard, some of whose systems were affected by the issue.
Problems emerged quickly after the update was released on Friday, with users posting photos on social media of their computers showing an error message and a blue screen of death, known in the industry as the “Blue Screen of Death.”
Patrick Wardle, a security researcher who specializes in studying threats to operating systems, said his analysis had identified the code that caused the outage.
He said the problem with the update was with “files that contain either configuration information or signatures — code that detects certain types of malicious code or malware.”
“It’s very common for security products to update their signatures once a day because they are continually monitoring for new malware and want to ensure their customers are protected from the latest threats,” he said.
“The frequency of updates is probably why[CrowdStrike]didn’t test it as much,” he said.
It’s unclear how the flawed code got into the update, or why it wasn’t detected before it was released to customers.
“Ideally, they should have rolled it out to a limited number of people first,” said John Hammond, principal security researcher at Huntress Labs. “That’s a safer approach to avoid the chaos that we saw.”
Other security companies have seen similar incidents in the past: In 2010, McAfee released a buggy antivirus update that shut down hundreds of thousands of computers.
But the global impact of the outage reflects CrowdStrike’s dominance: More than half of the Fortune 500 companies and many government agencies, including the Cybersecurity and Infrastructure Security Agency, the top U.S. cybersecurity agency, use its software.
