While a senior CrowdStrike official issued blanket apologies to House lawmakers at a Sept. 24 subcommittee hearing for the widespread Windows outages sparked on July 19 by a faulty software update from the company, leaders of the House Homeland Security Cybersecurity and Infrastructure Protection Subcommittee positioned the CrowdStrike outage – and process improvements the company is undertaking to avoid any repeat performance – also as a teachable moment for all concerned.
Rep. Mark Green, R-Tenn., who chairs the full Homeland Security Committee, said to open the hearing that “a global IT outage that impacts every sector of the economy is a catastrophe that we would expect to see in a movie,” and “something that we would expect to be carefully executed by a malicious and sophisticated nation-state actor.”
“To add insult to injury, the largest IT outage in history was due to a mistake,” the chairman continued. “In this case, CrowdStrike’s Content Validator used for its Falcon Sensor did not catch a bug in a channel file. It also appears that the update may not have been appropriately tested before being pushed out to the most sensitive part of a computer’s operating system. This caused about 8.5 million devices to crash.”
“Mistakes can happen,” Rep. Greene said, adding, “however, we cannot allow a mistake of this magnitude to happen again.”
“The good news is that since this was not due to a cyberattack, we can learn from this incident,” he said, adding that the Sept. 24 hearing was both timely “because we now have two months of information to understand exactly what happened,” but also overdue “because we had hoped to give Americans the answers they deserve much sooner, given the extent of the outage.”
Rep. Green recounted a comment last month from Cybersecurity and Infrastructure Security Agency (CISA) Director that the CrowdStrike outage was a “a useful exercise – a dress rehearsal for what China may want to do to us.” Rep. Green said at the hearing, “we look forward to working with you to make sure we never make it to opening night.”
Subcommittee Chairman Rep. Andrew Garbarino, R-N.Y., hit on similar themes in his opening statement, including how adversaries are viewing the CrowdStrike outage.
“The sheer scale of this error was alarming,” the congressman said. “If a routine update could cause this level of disruption, just imagine what a skilled and determined nation-state actor could do.”
“We cannot lose sight of how this incident factors into the broader threat environment,” Rep. Garbarino said. “Without question, our adversaries have assessed our response, recovery, and true level of resilience.”
“However, our enemies are not just nation-states with advanced cyber capabilities,” he continued. “They include a range of malicious cyber actors who often thrive in the uncertainty and confusion that arise during large-scale IT outages.”
In his prepared testimony for the hearing, Adam Meyers, who is senior vice president of counter adversary operations at CrowdStrike, offered apologies for the July outage and elaborated on the list of remedies the company has since undertaken to avoid a repeat.
“I am here today because, just over two months ago, on July 19, we let our customers down,” Meyers told members of the subcommittee. “As part of regular operations, CrowdStrike released a content configuration update for the Windows sensor that resulted in system crashes for many of our customers.”
“On behalf of everyone at CrowdStrike, I want to apologize. We are deeply sorry this happened and are determined to prevent it from happening again,” he said.
“More broadly, I want to underscore that this was not a cyberattack from foreign threat actors. The incident was caused by a CrowdStrike rapid response content update,” he said.
Since then, Meyers said, the company has “taken steps to help ensure that this issue cannot recur,” including introducing new validation checks for updates, and improving testing procedures to “cover a broader range of scenarios.”
In addition, Meyers said CrowdStrike has provided customers with additional controls over configuration updates to their systems. He said it has also put in place a system in which configuration information is released more gradually across “increasing rings of deployment,” and explained that is allowing the company “to monitor for issues in a controlled environment and proactively roll back changes if problems are detected before affecting a wider population.”
“We have added additional runtime checks to the system, designed to ensure that the data provided matches the system’s expectations before any processing occurs,” Meyers testified. “We are also working to further enhance our safeguards for validation and quality assurance, including by implementing more granular controls,” and using third-party reviews to compile “end-to-end quality control and release processes reviews,” he said.
In a separate report issued on Sept. 23, the Government Accountability Office (GAO) weighed in on the July CrowdStrike outage, with a focus not only on the incident itself but also on the need for Federal agencies and the private sector to tighten up their cooperation in response to cyber threats and incidents.
Since GAO put a series of cybersecurity concerns on its high risk list in 2018, “the federal government has also taken steps to improve the response to such incidents,” the watchdog agency said, adding, “but there is still work to be done to improve the way cyber incidents are managed and mitigated.”
“Although the CrowdStrike crash was caused by human error and not a cyberattack, it highlights similar vulnerabilities we saw during the SolarWinds attack in 2019,” GAO said in its new report. “In that event, instead of attacking systems directly, malicious actors targeted system support software” that was widely used by Federal agencies to monitor network activity and manage network devices, it said.
