The T-Mobile DDoS Attack That Wasn’t

Yesterday, the internet was atwitter with rumors of a massive distributed denial of service (DDoS) attack against major US carriers. You might have seen scary screenshots from one of the many so-called “pew-pew maps,” like this one:

 

T-Mobile was a focus of the rumors, as its own customers were complaining of services down.

The outage was reason for concern because there actually has been a new development in infrastructure weaponization, but before we get into that, let’s finish up with yesterday’s drama. T-Mobile publicly referenced only ongoing technical issues, but generic statements like this one are actually protocol during DDoS attacks, so it wasn’t necessarily an outright denial.

In spite of all of these indicators, and the recent development in DDoS techniques (wait for it), there were three good reasons to be skeptical about the T-Mobile outage being a DDoS attack:

  1. A DDoS attack so large as to disrupt text, data, and voice services at multiple telecom carriers would set off alarm bells all over the internet, as happened during the Mirai attacks in 2016.
  2. Over the years, there have been many, many cases where a routine configuration outage was assumed, by both internal and external parties, to have been the result of a DDoS attack. Most of the time, someone misconfigured DNS or let a certificate expire.
  3. It’s human nature to want to ascribe to malice what is actually just bad luck or bad weather. Malice makes for better headlines, feeds conspiracy theorists, and takes the good guys off the hook (a bit).

It’s looking like the T-Mobile outage wasn’t a DDoS attack, just an unsuccessful upgrade or config change with cascading failures on T-Mobile’s part. Other parties, including Brian Krebs, are coming to a similar conclusion.

As of this writing, T-Mobile hasn’t come forth with a root cause analysis, and they may never, but if past experience is a guide, you can bet dollars to donuts it’s something to do with DNS or certificates. Recall that the massive O2 outage, affecting millions of users in the UK in 2018, turned out to be not a DDoS attack but just an expired SSL certificate.

Either way, there is a lesson to be learned here, for network operators, newly minted CISOs, and the press: If someone jumps up with a claim of DDoS when there’s an outage, look first for the much more pedestrian root cause: human error. This should apply for outages at your own company, as well. A second lesson might be a reminder that the internet is largely composed of baling wire and bubble gum, so should anyone be surprised when the proverbial woodpecker comes calling periodically?

NXNSAttack

But speaking of denial-of-service attacks, did you hear about last month’s hipster DDoS attack? It’s based on a really obscure threat vector that you’ve probably never heard of before.

In May, three Israeli researchers, Lior Shafir, Yehuda Afek, and Anat Bremler-Barr, publicly debuted a DNS amplification attack called NXNSAttack. The attack has its own website but not its own logo, theme song, or cryptocurrency like the TLS vulnerabilities do. Read the paper for the gory details on NXNSAttack if you like; the upshot is that the attack had the theoretical power to amplify an attacker’s initial signal by a factor of 1,620. That’s three orders of magnitude and, if properly weaponized, could cause enormous havoc.

NXNSAttack has a happy ending in that the researchers followed the responsible disclosure script, warning the affected vendors (Microsoft, Google, and ISC BIND of course, and others) and giving everyone time to get their ducks in a row, as it were, before public notice. So, in theory, the internet has been properly inoculated against NXNSAttack already, but the T-Mobile false alarm had me worried for a minute.

Johannes Ullrich of SANS was recently interviewed on The CyberWire podcast about the DNS-reflection honeypot his team has been working on. Tying all of this together, in theory, the honeypot could have detected if the NXNSAttack was being used on DNS servers to attack T-Mobile yesterday. Mind blown — that’s some extra cool baling wire right there.

Now Tech: DDoS Mitigation Solutions, Q2 2020

And lastly, not to be lost amid all this drama, in April, we published the Forrester report, “Now Tech: DDoS Mitigation Solutions, Q2 2020.” The report groups 22 different DDoS vendors by size and functionality. Significantly, in this report, for the first time, five of the largest cloud service providers field offerings: AWS, Azure, GCP, Alibaba, and Tencent.

Getting back to my lame hipster DDoS joke (yes, these come to me after a beer on my porch): Regardless of whether the T-Mobile outage was or wasn’t a DDoS attack, most organizations are going to need DDoS protection for years to come; the problem isn’t just going away. As the NXNSAttack technique demonstrates, new vectors are still being found. So if your organization finds itself handling a storm of DNS subdomain resolutions that no one requested and you find yourself in the market for a new DDoS umbrella, now you know what report to look at to see which one might be right for you.