What Was Wrong With BT Broadband This Week?
ANALYSIS: Was your connection down this week? BT broadband suffered from two separate data centre outages in two consecutive days. Here’s what happened
If you’re one of the more than eight million subscribers to BT broadband, you may have noticed a few issues with your connection this week.
Two separate outages affected as many as one tenth of the company’s customer base, incidents that naturally caused disruption for the businesses that are reliant on the country’s largest internet service provider (ISP).
@bt_uk @btcare bt broadband down again, this is costing my business hundreds of pounds for every hour down #grindingtheukeconomydown
— motherofthebride (@moboutfits) July 21, 2016
Problems with #btbroadband is still affecting contact with @ParkinsonsUK home-based staff today. @bt_uk is working on it – pls bear with us. — Parkinson’s UK (@ParkinsonsUK) July 21, 2016
Services were affected on both Wednesday 20 July and Thursday 21 July, but what exactly happened?
Day 1
Up to ten percent of BT customers were left without broadband for a number of hours following a loss of power at a data centre in London’s Docklands between 07:55 and 08:17 BST.
The Equinix-owned Telecity LD8 data centre houses equipment for the London Internet Exchange (LINX), which is a BT supplier and it was this that caused the disruption to the broadband network. Power was restored to equipment by 08:17 apart from a single device and full power was restored at 09:15.
The root cause of the issue was a faulty UPS (uninterrupted power supply) system, Equnix said, adding it was working with customers to minimise the impact. BT engineers were still working to fix the problems at 11:07am and service was restored just after midday.
Day 2
But then, almost exactly 24 hours later, BT customers were reporting further disruption. Astonishingly, a second power outage at a different data centre from a different supplier was behind it. This time it was Telehouse, which blamed the tripping of a circuit breaker with its Telehouse North facility.
“Telehouse can confirm that an equipment failure led to service disruption within its Telehouse North data centre in London this morning,” said the data centre firm. “Regrettably, this incident resulted in the loss of service for a specific and limited group of customers within the building for which we apologise. Telehouse engineers moved swiftly to rectify the problem and all services were fully restored by 1.30pm.
This time, up to five percent of BT’s user base was affected, causing the ISP to redirect traffic while its engineers took action.
Impact
As of Thursday morning the connection issues relating to the two incidents had been fixed, but it’s possible BT (and Plusnet) will suffer from reputational damage.
Outages happen for all providers and neither case was BT’s direct fault but it is unlikely most customers will be aware of the root causes. After all, there is still confusion among the public about Openreach’s role in the installation and repair of telephone lines.
The most obvious takeaway is hardly Earth shattering: business are extremely reliant on their broadband connection. But it’s also remarkable how two separate power failures can take place on two consecutive days and create such mayhem for the UK’s Internet infrastructure.
