For years, there have been concerns of the Chinese government building backdoors and spying capabilities in to phones and hardware built on their soil, and now it seems those concerns are coming to head with a recent Bloomberg story that alleges that Chinese government agents installed thousands of spy chips into servers used by Amazon, Apple, and the US government. How true are these allegations, are our servers safe? In this article, we’ll attempt to figure out what—if anything—happened, and how IT pros should react.
The Story
So by now, you’ve probably read the Bloomberg article, The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies, and if you haven’t, you should go read it. But just in case you’re lazy, here’s a recap:
According to Bloomberg, Agents working for the Chinese government infiltrated Super Micro, one of the world’s biggest suppliers of server motherboards, and implanted some motherboards built in China with tiny chips—smaller than a grain of rice— that had the capability to alter the server’s operating system so that it would accept modifications, exfiltrate information, and contact an outside computer for further instructions from attackers. According to Bloomberg, these chips were initially discovered by Amazon during a routine audit of a prospective acquisition’s technology stack, and have been reported to the FBI, where an ongoing investigation was opened.
The Bloomberg article is sparse on technical details, but it seems as though the tiny implants were connected to the baseboard management controller (BMC), a component that gives systems administrators the ability to remotely access machines, and could intercept and modify signals from it, which would give it sufficient privileges to backdoor the server. Here’s how Bloomberg put it:
" … The implants on Supermicro hardware manipulated the core operating instructions that tell the server what to do as data move across a motherboard… this happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects."
The attack originated from four subcontractors for SuperMicro, and was highly targeted, affecting blue-chip companies like Apple and Amazon, as well as banks and government contractors.
The report relied on 17 unnamed sources, whom Bloomberg claims are current or former insiders at Apple, Amazon, and the FBI.
The Implications
Needless to say, if true, this is huge. It doesn’t take much to see the wide spread implications of such an attack. First let’s look at the security implications of the attack. According to Bloomberg, this attack affected Apple, Amazon, Department of Defense data centers, Navy warships, and the CIA’s drone operations. So, from an OPSEC perspective, the military is insecure, unless they find a way to find and mitigate every single one of these chips. Businesses in direct competition with Chinese companies are also compromised. Not good.
What’s more, a supply-chain attack at this scale would not only give China the ability to spy on thousands of servers, both in critical business and government facilities, it would also call into question the security of our entire tech supply-chain, which is largely based in China. The idea that we can trust out supply chain has been an article of faith in the tech world for decades. It’s long been accepted that China could pull off this sort of espionage, but it was taken for granted that they wouldn’t. The stakes are simply too high. If U.S. tech companies (or consumers) were spooked by this story, and decided to pull manufacturing from China, it would be a gut-punch to the Chinese economy, to Silicon Valley, and to the tech economy in general—if not the global economy.
If this story turns out to be true, there’s no doubt that there will be criminal and congressional investigations, and likely an elevation of the President Trump’s ongoing trade war… but if is the operative word, here’s why:
The Response: Deny, Deny, Deny
When Apple and Amazon caught wind of Bloomberg’s report, they vehemently denied that there was any truth to the tale of the hardware hack.
In their response to the story, Apple cast doubt on Bloomberg’s reporting, writing that “Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.”
The response went on to say that Apple has “never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server,” and that Apple “never had any contact with the FBI, or any other agency about such and incident,” and that they are not aware of any investigation by the FBI.
Apple even went as far as doubling down on their denials by sending an open letter to the US House and Senate Commerce Committees on Monday, October 8th.
For their part, Amazon was equally unequivocal.
“[The Bloomberg Story] is untrue. At no time, past or present, have we found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon Systems. Nor have we engaged in an investigation with the government,” wrote Stephen Schmidt, Amazon’s CISO.
SuperMicro, the server company at the center of the scandal was also on the denial train, telling Bloomberg that “while we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue.”
While it’s common for companies to deny this sort of data leak, or even keep it under wraps until they’re ready to address the issues, these denials struck me as unusual. They’re very resolute—to the point that, if they were found to be lying, Apple and Amazon could be sued for securities fraud. If these companies were lying, you typically wouldn't expect them to double down like this.
I have to say, this is all really bizarre. The Bloomberg story is very detailed, citing documents and inside sources. But the company denials are also detailed and emphatic. You don't often see the latter when a company is trying to hide something or be coy. https://t.co/qjA1TFKzZ3
— Kim Zetter (@KimZetter) October 4, 2018
China, in a statement that should be taken with a grain of salt, characterized itself as a “resolute defender of cybersecurity” that is “also a victim” of supply chain safety concerns. Of course, China doesn’t have a history as a “resolute defender of cybersecurity,” but the denial is in line with what the supposedly affected companies have said. Finally, in what was perhaps the most damning statement for Bloomberg’s article, both the US and UK national security apparatuses responded over the weekend, each saying, in essence, “we don’t know for sure what happened, but we believe Apple and Amazon.” Here’s what the US Department of Homeland Security’s response looked like: “The Department of Homeland Security is aware of the media reports of a technology supply chain compromise. Like our partners in the UK, the National Cyber Security Centre, at this time we have no reason to doubt the statements from the companies named in the story.”
But Wait! There’s More! Bloomberg Makes Claims of Another Hardware Hack
So, days after the initial report, the legitimacy of Bloomberg’s claims are still being argued, with both sides doubling down on their claims. Now, Bloomberg has released another, related story, which claims that an unnamed “major telecommunications company” discovered a similar hardware hack in SuperMicro components on their network, and removed those components in August of this year. Bloomberg claims to have seen “documents, analysis, and other evidence” that backs up their claims in this new story, and this time around, they even have a named source: Yossi Appleboum, CEO of Sepio Systems, a cybersecurity firm hired to audit security at the unnamed telecom company. According to Appleboum, he first noticed “unusual communications” sent from a Super Micro server, and eventually discovered “an implant built into the server’s Ethernet connector,” which Applebaum says was the culprit. This is, of course, a different compromise than the one purported by the original article, but the effect is the same, according to Bloomberg—a secret part made at a Super Micro factory and designed to exfiltrate data invisibly. On the topic of the DHS’s denial of supply chain attacks, Appleboum told Bloomberg that “he was told by Western intelligence contacts that the device was made at a Super Micro subcontractor factory in Guanzhou.” If true, this would mean that the DHS did, in fact, know of hardware hack-based supply chain attacks. So how has the Government responded? FBI Director Christopher Wray touched on the topic on Wednesday during a hearing in front of the Senate Homeland Security Committee, where he told senators to “be careful what you read.”
The Technical Details Cast Doubt
If that series of denials and rebuttals muddies the story, things get even more unclear when you start to look for technical details of the alleged supply chain attack. At the core of the story is the claim that these tiny extra components—smaller than a grain of rice—have enough capability to carry out the malicious actions required for the hack to be effective. This tiny chip would have to be able to alter a server’s OS so that it could accept modifications, and also call back to an outside computer for further instructions. Sounds complicated, right? It is. In fact, it may be too complicated to be a reality—or at least to be a practical reality—and this is where some people in the security community are taking issue with the Bloomberg story. One researcher who was a named source in the original Bloomberg story, has even cast doubt on the technical evidence for the hack. FitzPatrick, who is the founder of Hardware Security resources, LLC, told hosts of the Risky Business Podcast that the hardware backdoor described in Bloomberg’s story “didn’t make sense.” FitzPatrick expanded on this comment on his Twitter account, where he wrote that there are “plenty of software vectors for exploiting a system. None of them require silicon design, hardware prototyping, or manufacturing processes, and none of them leave behind a physical item once they’re implanted.” For a more detailed version of FitzPatrick’s take, check out his article on the topic here.
Installing malicious software on 10,000 systems is a system management problem. Installing malicious hardware on 10,000 systems starts out as an HR problem and moves on from there. It’s just not scalable. — Joe Fitz (@securelyfitz) October 4, 2018
FitzPatrick also took aim at Bloomberg’s follow up article, and it’s singular source, Mr. Appleboum at Sepio Systems:
I can only hope that this isn't the same document provided to bloomberg. pic.twitter.com/NrnhOCCohg
— Joe Fitz (@securelyfitz) October 9, 2018
The Verdict, And What Can Be Done About It
So this is probably what you were waiting for, and I’m sorry to say, the verdict is still out.
On one hand, it’s a lot of smoke for no fire, and you wouldn't expect a world-class news outlet like Bloomberg to publish such bombshell allegations without submitting them to the most rigorous fact-checking possible. And even if they did, you wouldn't expect them to double down with another story making the same claims just days later.
But on the other, things just aren’t adding up for Bloomberg’s story.
Starting with the technical details—yes something like this is technically possible, but it would take a lot of work, both on the technical end, and in terms of sheer manpower. All that, when the same effect could very likely be achieved with a more traditional software hack targeting the BMC.
We also have extremely direct denials from the tech companies involved—Apple even sent a letter to congress. If all of those denials turn out to be lies, those companies could face major legal repercussions ranging from securities fraud to GDPR noncompliance.
And then there are the real-world implications of Bloomberg's story. The report had a huge impact on stock markets—Super Micro’s stock dropped 50% in the first day, while Amazon and Apple’s stocks dropped roughly 2 percent. Those stocks have since recovered, but it’s enough to make you wonder: did Bloomberg accidentally report on a disinformation campaign?
Whatever the case, one thing is certain: Bloomberg's story made hardware hacks top-of-mind for IT pros across the world. So, with that in mind, how would you discover such an attack if it was on your network? I'll let @MalwareJake handle that one:
Most of us obviously don't have the capabilities to inspect our motherboards for rogue chips. But we do have the ability to monitor network traffic. If this doesn't speak to the urgent need for full network monitoring, I don't know what will.https://t.co/RHa9QZKzjZ
— Jake Williams (@MalwareJake) October 4, 2018
The most obvious step to discovering this attack would be from one of your machines. If you see a machine sending messages to an IP address you don't recognize, that's a sure sign that something's wrong. With a full-fledged network monitoring tool, you can easily watch ingress and egress traffic, and set alerts for when things aren't right. Some tools, such as WhatsUp Gold, can even identify traffic coming to and from the dark web. That's not to say that Network Monitoring should be your only security tool, but it's a start.