Misconfigured CenturyLink database caused global internet outage

American ISP CenturyLink suffered an incident that caused an internet outage felt all over the world this weekend.

An incorrect Flowspec announcement in the company’s data center near Ontario, Canada is said to have caused a technical problem that affected the service of a range of other organizations.

Businesses including Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu, Duo Security, Imperva, NameCheap, OpenDNS were all affected by the outage.

In total, there was a drop in global internet traffic of approximately 3.5 percent, according to web performance firm Cloudflare. 

As explained by Cisco, the issue was linked with the BGP (Border Gateway Protocol) flow specification (flowspec) feature, which “allows you to rapidly deploy and propagate filtering and policing functionality among a large number of BGP peer routers to mitigate the effects of a distributed denial-of-service (DDoS) attack over your network.”

Analysis suggested CenturyLink had looped its entire network by announcing new BGP routes and then dropping all of them through the misconfigured Flowspec rule.

To fix the issue, CenturyLink was forced to reset all its gear and start with clean BGP routing tables, which is said to have taken almost seven hours.

This is not the first time CenturyLink experienced an outage on such a large scale. In late 2018, a malfunctioning network card took phone and internet services offline for the majority of users, including 911. Some users waited two days for services to be restored.

Source Article