August 21, 2020

1651 words 8 mins read

Network Outages Go Global

Network Outages Go Global

On August 30, CenturyLink experienced a major network outage that lasted for over five hours and disrupted CenturyLink customers nationwide as well as many other networks. What was unique about the outage was the scope of the disruptions as the outage affected video streaming services, game platforms, and even webcasts of European soccer.

This is an example of how telecom network outages have ex

panded in size and scope and can now be global in scale. This is a development that I find disturbing because it means that our telecom networks are growing more vulnerable over time.

The story of what happened that day is fascinating, and I’m including two links for those who want to peek into how the outages were viewed by outsiders who are engaged in monitoring Internet traffic flow. First is this report from a Cloudflare blog that was written on the day of the outage. Cloudflare is a company that specializes in protecting large businesses and networks from attacks and outages. The blog describes how Cloudflare dealt with the outage by rerouting traffic away from the CenturyLink network. This story alone is a great example of modern network protections that have been put into place to deal with major Internet traffic disruptions.

The second report comes from ThousandEyes, which is now owned by Cisco. The company is similar to Cloudflare and helps clients deal with security issues and network disruptions. The ThousandEye report comes from the day after the outage and discusses the likely reasons for the outage. Again, this is an interesting story for those who don’t know much about the operations of the large fiber networks that constitute the Internet. ThousandEyes confirms the suspicions that were expressed the day before by Cloudflare that the issue was caused by a powerful network command issued by CenturyLink using Flowspec that resulted in a logic loop that turned off and restarted BGP (Border Gateway Protocol) over and over again.

It’s reassuring to know that companies like Cloudflare and ThousandEye can stop network outages from permeating into other networks. But what is also clear from the reporting of the event is that a single incident or bad command can take out huge portions of the Internet.

That is something worth examining from a policy perspective. It’s easy to understand how this happens at companies like CenturyLink. The company has acquired numerous networks over the years, from the old Qwest network up to the Level 3 networks, and has integrated them all into a giant platform. The idea that the company owns a large global network is touted to business customers as a huge positive but is it?

Network owners like CenturyLink have consolidated and concentrated the control of the network to a few key network hubs controlled by a relatively small staff of network engineers. ThousandEyes says that the CenturyLink Network Operation Center in Denver is one of the best in existence, and I’m sure they are right. But that network center controls a huge piece of the country’s Internet backbone.

I can’t find where CenturyLink ever gave the exact reason why the company issued a faulty Flowspec command. It may have been used to try to tamp down a problem at one customer or have been part of more routine network upgrades implemented early on a Sunday morning when the Internet is at its quietest. From a policy perspective, it doesn’t matter what matters is that a single faulty command could take down such a large part of the Internet.

This should cause concerns for several reasons. First, if one unintentional faulty command can cause this much damage, then the network is susceptible to this being done deliberately. I’m sure that the network engineers running the Internet will say that’s not likely to happen, but they also would have expected this particular outage to have been stopped much sooner and easier.

I think the biggest concern is that the big network owners have adopted the idea of centralization to such an extent that outages like this one are more and more likely. Centralization of big networks means that outages can now reach globally and not just locally like happened just a decade ago. Our desire to be as efficient as possible through centralization has increased the risk to the Internet, not decreased it.

A good analogy for understanding the risk in our Internet networks comes by looking at the nationwide electric grid. It used to be routine to purposefully allow neighboring grids to automatically interact until it was obvious after some giant rolling blackouts that we needed firewalls between grids. The electric industry reworked the way that grids interact, and the big rolling regional outages disappeared. It’s time to have that same discussion about the Internet infrastructure. Right now, the security of the Internet is in the hands of few corporations that stress the bottom line first and which have willingly accepted increased risk to our Internet backbones as a price to pay for cost efficiency. Written by Doug Dawson, President at CCG ConsultingFollow CircleID on TwitterMore under: Access Providers, Networks

Author: Doug Dawson

Date: 2020-11-13


Donuts Acquires Afilias (2020-11-19) Donuts and Afilias announced today that Donuts is acquiring Afilias in a deal that is expected to close in December 2020 for an undisclosed amount The combined entities will support over 25 million domain names spanning well over 400 TLDs The deal will not include certain Afilias businesses such as the mobile software and registrar businesses which will remain with Afilias original group of invest..
A Look at the Big Guys - Putting the Telecom Sector Into Perspective (2020-12-01) You cant put the telecom sector into perspective without looking at the performance of the biggest players in the industry The pandemic has been an interesting year for both big ISPs and telecom vendors Smaller ISPs should care about big ISP performance for many reasons For many smaller companies the big companies are the competition and the big providers strength or weakness can foretell stiffene.. A Look at the Big Guys - Putting the Telecom Sector Into Perspective
What Will 6G Look Like in a Geopolitical Divided World (2020-11-27) A recent session of the Australian Computer Society ACS and the Australian Smart Communities Association ASCA masterclass on 5G featured Dr Ian Oppenheimer the NSW Governments Chief Data Scientist and one of the countrys most respected experts in radio technology Ian is also the Chair of the scientific advisory board of the 6G Flagship organization a global research organization based in Finland T.. What Will 6G Look Like in a Geopolitical Divided World
What Does an Administration Change Mean for the FCC? (2020-11-17) Just as the last change in administration changed the course of the FCC so will the swing back to a Democratic administration If youve been reading me for a few years you know I am a big believer in the regulatory pendulum Inevitably when a regulatory agency like the FCC swings too far in any direction its inevitable that it will eventually swing back the other way If I had to characterize the cur.. What Does an Administration Change Mean for the FCC?
Authenticated Resolution and Adaptive Resolution: Security and Navigational Enhancements to the DNS (2020-11-19) The Domain Name System DNS has become the fundamental building block for navigating from names to resources on the internet DNS has been employed continuously ever since its introduction in 1983 by essentially every internet-connected application and device that wants to interact online Emerging from an era where interconnection rather than information security was the primary motivation DNS has g..
The End of Broadband Networks as We Know Them (2020-11-24) A group of major telecommunications companies Vodafone BT Telefonica and Deutsche Telekom recently announced something a bit unexpected In the Open BNG Operator Position Paper they call for a fundamental industry-wide change to the way broadband networks are built Specifically they advocate for collaboration throughout the telco world with a convergence around open broadband network gateways BNGs .. The End of Broadband Networks as We Know Them
Every Internet Governance Stakeholder Has a Role to Play in the Online Health Debate (2020-10-26) Note: This article is part of the preparation for Workshop #116 of the UNs 2020 Internet Governance Forum: Pandemics Access to Medicines: A 2020 Assessment which will take place in November 16 from 14:00 to 15:30 UTC Please find more information HERE Much has been discussed in relation to the impacts of the COVID-19 pandemic on most aspects of human life but the dialogue around the long-term reper..
U.S. Military Is Buying Location Data of People Around the World Through Ordinary Apps (2020-11-17) The US military is buying the granular movement data of ordinary people worldwide harvested from innocuous-seeming apps reports Motherboard Among the apps connected to this type of data sale is a Muslim prayer and Quran app with more than 98 million downloads worldwide Other apps include a Muslim dating app a Craigslist app an app for following storms and other small-type utility apps The report r..
An Investigative Analysis of the Silent Librarian IoCs (2020-11-12) The Silent Librarian advanced persistent threat APT actors have been detected once again as the academic year started in September With online classes increasingly becoming the norm the groups phishing campaigns that aim to steal research data and intellectual property could have a high success rate Dozens of phishing domain names have been reported although some may have already been taken down S..
Backlash Over Potential Firing of U.S. Election Cybersecurity’s Top Official (2020-11-17) CISA Director Chris Krebs on the ongoing effort to secure the 2020 election and rumor control The Electronic Frontier Foundation EFF along with over forty other cybersecurity experts and organizations are urging the White House to keep politics out of securing this months election in the US The concern arose following a Reuters report that the top US cybersecurity official Christopher Krebs Homela..