CrowdStrike Bug Sparks Global Cyber Outage: Error in a single line of C++ code triggers global tech meltdown [Update]
Daniel Levi
Posted On July 19, 2024
For many hours today, the internet stood still as a domino effect of computer outages plunged vast regions of the globe into digital darkness. Airlines, financial institutions, online communications, and countless other services sputtered and died, leaving the world in disbelief at the collapse of what seemed like an invincible network.
The shocking culprit? According to a bombshell analysis by cybersecurity firm Crowdstrike, it all traced back to a programmer error in a single line of code—a seemingly innocuous glitch in the ancient, memory-hungry language of C++.
In a thread on X, Zach Vorhies claimed that a global tech meltdown was triggered by an error in a line of C++ code. For those unfamiliar, Vorhies is a former Google employee who famously leaked 950 pages of internal documents to the DOJ, revealing details about Google’s censorship practices.
“It was a NULL pointer issue due to the memory-unsafe nature of C++,” Vorhies explained in his X post.
How an Error in a Single Line of C++ Code Caused the Microsoft Global Outage
According to Vorhies, during a software update, Crowdstrike’s code attempted to access “memory address 0x9c (or 156),” an invalid memory region, subsequently triggering the worldwide outage. Below is a screenshot of the stack trace dump Vorhies shared on X.
CrowdStrike stack trace dump
Crowdstrike Analysis:
It was a NULL pointer from the memory unsafe C++ language.
Since I am a professional C++ programmer, let me decode this stack trace dump for you. pic.twitter.com/uUkXB2A8rm
— Zach Vorhies / Google Whistleblower (@Perpetualmaniac) July 19, 2024
“So why was memory address 0x9c being accessed?” Vorhies asked. “The answer is programmer error. In C++, which CrowdStrike uses, address 0x0 is a special value that signifies ‘there’s nothing here’—trying to access it results in a fatal error,” he explained.
C++ programmers are expected to handle this by checking for null values when passing objects around. Typically, you might see code like this: string* p = get_name(); if (p == NULL) { print(“Could not get name”); }.
In the stack dump of CrowdStrike’s code, Vorhies noted that the program attempted to read memory address 0x9c, which translates to the numeric value 156.
Vorhies further explained that the issue stemmed from a programmer’s failure to validate the object’s validity before accessing its member variables. The calculation NULL + 0x9C = 0x9C = 156 points to an invalid memory region.
“So what happened is that the programmer forgot to check that the object it’s working with isn’t valid, it tried to access one of the objects member variables…”
In this stack dump you see that it's trying to read memory value 0x9c. In human numbers, this is the value 156.
So what happened is that the programmer forgot to check that the object it's working with isn't valid, it tried to access one of the objects member variables…
— Zach Vorhies / Google Whistleblower (@Perpetualmaniac) July 19, 2024
Blue Screen of Death
Blue Screen of Death
This error was particularly problematic because it involved a system driver with privileged access to the computer. To protect system integrity, the operating system had no choice but to crash immediately.
“This is what caused the Blue Screen of Death,” Vorhies said. “While non-privileged code can usually recover from a crash by terminating the program, system drivers can’t. When your computer crashes, it’s often due to a failure in the system drivers.”
Vorhies noted that if the programmer had performed a NULL check or used modern tooling designed to catch such issues, the error might have been avoided. Unfortunately, the flaw slipped through the cracks, made it into production, and was then pushed out as a mandatory update by CrowdStrike. Oops!
If the programmer had done a check for NULL, or if they used modern tooling that checks these sorts of things, it could have been caught. But somehow it made it into production and then got pushed as a forced update by Crowdstrike… OOPS!
— Zach Vorhies / Google Whistleblower (@Perpetualmaniac) July 19, 2024
Recommendations for Microsoft and CrowdStrike
Going forward, Vorhies recommended that Microsoft implement more robust policies for rolling back defective drivers, rather than pushing risky updates directly to customers. He also suggested that CrowdStrike should elevate its code safety officer role to focus on integrating code sanitization tools that can automatically catch such issues. Additionally, Vorhies indicated that CrowdStrike might consider rewriting their system driver from C++ to a more modern language like Rust, which inherently avoids this kind of problem.
Update – July 21, 7:30 PM EST:
New Doubts on Zach Vorhies’ Crowdstrike C++ Code Analysis
Fresh insights have emerged, casting doubt on Zach Vorhies’ analysis of the C++ code that allegedly caused the cyber outage on Friday. Tavis Ormandy, a vulnerability researcher at Google, highlighted major flaws in Vorhies’ conclusions in a thread post on X.
Ormandy pointed out that Vorhies’ analysis was based on a “screenshot of !analyze -v output.” Furthermore, he noted that Vorhies confused the terms “stack trace” and “minidump,” and only reviewed the decoded exception record, hastily concluding that “it was a NULL pointer.”
It is a plausible explanation, 0x9c is not NULL, but dereferencing near-NULL addresses can have the same root cause. He explains that the code was reading a field at offset 156 from a NULL object pointer. 3/n pic.twitter.com/p4ucDWoRxw
— Tavis Ormandy (@taviso) July 20, 2024
Ormandy stated that the CrowdStrike C++ code analyzed by Vorhies doesn’t align with the version tested on Godbolt (https://godbolt.org/z/sdz4PGxx).
“The code is either more complicated, or his hypothesis is incorrect. There is a way to check, he could type `u` (unassemble) into kd and examine the surrounding code,” Ormandy explained.
https://twitter.com/taviso/status/1814762312211046703
Ormandy also clarified that Vorhies’ version of the faulting module includes the bytes 45 8b 08 at csagent+0xe35a1. After locating that version in VirusTotal, Ormandy noted that there is, in fact, a NULL check (test r8, r8; jz) immediately before the dereference, proving Vorhies’ theory wrong.
Ormandy explained that the code reads pointers from a table in a loop, and some of these pointers are invalid. It is possible that an error parsing a configuration file left some entries uninitialized, with one just happening to be 0x9c. This is just a theory, but at least it fits the facts.
Ormandy further referenced another instance observed by Patrick Wardle, Founder of the Objective-See Foundation.
https://twitter.com/taviso/status/1814762315587461495
“Here is the same crash seen by Patrick, but in his case, the entry was 0xffff9c8e00000008a, nowhere near NULL! If this is uninitialized data, it might have been fine during testing, which is why CrowdStrike didn’t catch it.”
Ormandy concluded his post with a touch of humor, noting the irony that Patrick Wardle, who genuinely knows what he’s doing, cautiously prefaced his analysis with “(initial) details.” Meanwhile, Vorhies confidently declares himself a professional and garners 25k retweets.
So, is Vorhies wrong? It’s still too early to say. However, during an interview on the Today Show, CrowdStrike CEO stated that the global cyber outage was caused by “a single content update.” Watch!
Trending Now
