The July incident stemmed from a glitchy content update for a sensor product that is used to detect and block cyber attacks. — Photo by Shekai on Unsplash
Cybersecurity company CrowdStrike came before a congressional subcommittee this week to explain how its faulty software update temporarily bricked 8.5 million Windows machines around the world, and how it would work to ensure it doesn't happen again.
CrowdStrike Senior Vice President of Counter Adversary Operations Adam Meyers said the company is now taking more precautions when releasing updates. But he also pushed back against lawmakers who questioned if the company erred in implementing its update deep inside Windows operating systems. While some lawmakers argued that pushing updates directly to the operating system's kernel meant a mistake could have serious consequences, Meyers said this deep-level access is important if cybersecurity software is going to keep up against hackers.
The July incident stemmed from a glitchy content update for a sensor product that is used to detect and block cyber attacks. At the time, CrowdStrike didn't treat content updates as cautiously as it does code updates. In this instance, the validator tool used to check content updates missed finding a defect and marked the faulty update as good to go. At that point, CrowdStrike pushed it out to all its global customers simultaneously. This bad content update essentially instructed sensors to perform an impossible action, causing the struggling sensors to fail. The sensors operated at the kernel level of Windows operating systems, and so when the sensors failed, they caused the systems to fail, too.
"It's almost like if you think about a chessboard trying to move a chess piece to someplace where there's no square ... when [the kernel sensor] tried to process the rule, it was not able to do what the rule was asking it to do," Meyers said. "... The perfect storm was the content validator allowed the content configuration to go out to the sensor, and the sensor was not able to find the rule that it was looking for, causing the issue."
CrowdStrike is now applying the same level of safeguards to content updates as it does with code updates, "which is something that I don't believe to be the industry standard at this time," Meyers said.
That means its average 10 to 12 daily content updates no longer are pushed out to all users at the same time. Instead, CrowdStrike will first test and deploy its content updates internally, then issue them to customers who opt in to be early adopters. That's a choice likely to be taken by people who want to test the updates on their own systems and check for unexpected behaviors, Meyers said.
The next group of customers to receive updates are the "general availability" group. Then, other customers can opt to receive updates later or not at all, Meyers said. Customers might opt to hold out longer before updating critical and highly sensitive systems to reduce risks that an undiscovered glitchy update downs something essential. But anyone waiting must also keep in mind that cybersecurity landscape evolves rapidly, and updates are needed to respond to timely threat information.
Reps. Eric Swalwell (D-CA) and Andrew Garbarino (R-NY) also pressed Meyers on why CrowdStrike releases updates directly into the Windows operating system kernel, given that an error at that level could crash an entire system.
Garbarino cited other cybersecurity providers reporting that access to the kernel is not safe, while Swalwell said releasing updates in user mode would only risk crashing an application, not the system.
Meyers, however, said frequent content updates are essential to keeping up against threats and that he wasn't aware of industry standards or best practices for how to update particular operating systems.
Accessing the kernel allows cybersecurity software to have visibility into all activities on the operating system, enforce security rules and block potential tampering by threat actors trying to gain access to a system, Meyers said. Scattered Spider, for example – the social engineering-savvy cyber criminal group that hacked MGM Resorts – has been using techniques to elevate their access to the kernel to disable security rules, he said. That makes it important for defenders to have kernel access to block them. Meyers avoided suggesting any particular operating system's approach to the kernel and kernel access was better than the other. He said security products are designed for each specific operating systems' features.
Rep. Will Timmons (R-SC) questioned whether anything would be done to compensate end customers who suffered damages from the incident, like passengers who missed flights or organisations that couldn't do business, saying, "That's part of the conversation that we need to be having." – Government Technology/Tribune News Service
