We take a trip to AT&T’s national disaster recovery centre to learn how to get a devastated area back on its feet
When a natural disaster hits, it can often take years for a community to recover, as homes, jobs and even lives lost. A quick response is paramount in these situations, and can often make all the difference, especially in a remote location or one dependent on high levels of communication.
TechWeek Europe was invited to visit the global NDR (National Disaster Recovery) centre of American telecoms provider AT&T, situated not in the United States, but instead a much more rural location in England. There, the company stores and maintains all the resources it needs to get the communications infrastructure of a region up and running in as short a time as possible, actions that may well contribute towards saving lives.
AT&T has spent around $600 million on its NDR program throughout in its 20-year history, Justin Williams, the facility’s director, told TechWeek Europe. This includes over 320 pieces of equipment (including 90 trailers) stored throughout multiple sites, the majority of which are stored in the UK centre, which is situated close to many major transport hubs for quick and effective dispatch in case of an emergency, as well as being the base for many of the companies top engineers.
The NDR staff is made up of around 30 AT&T trained specialists, but also consists of around 70 volunteers from other areas within the company, with employees encouraged to sign up to be called upon in case of an emergency. Training is done in-house and honed in a major trials held around twice a year, with the company carrying out a full rollout of all its services to ensure everything is running smoothly.
The range of options open to the AT&T team is extremely impressive to say the least – our visit included a tour of most of the vehicles used in disaster areas, varying from two-person units to a full-scale mobile office, alongside high-powered infrastructure units which will provide everything needed to kick-start a blacked-out region.
AT&T says that almost 90 percent of its activities are pre-planned, but the company’s vehicles provide its workers with all the necessary tools to survive in remote locations, with everything needed to work and live in extreme conditions alongside built-in generators which allow the facilities to stay powered up.
The team has spent more than 130,000 working hours in the field over the last 22 years, with notable deployments in recent years including the 2010 Chilean earthquake, Hurricane Katrina and the impact of Super Storm Sandy in 2012.
In these situations, AT&T’s NDR team, which is on call 24 hours a day, 7 days a week, will look to deploy to the affected area as quickly as possible, with the company’s equipment either transported via road or by air – being designed especially for straightforward loading.
The company’s fleet of vehicles (pictured right) includes nine trailers dedicated to the recovery of the core, high-capacity routers that send and receive all of the network traffic from one network office to the next. These trailers can support data traversing at rates as high as 100GB/s, but when fully up to speed, the trailers can handle a capacity of over 15 Terabits per second.
These trailers, which Williams says are “100 percent ready, 100 percent of the time”, are able to handle large amounts of network traffic— from a city of smart phone users to high-volume data transfers between businesses, meaning they can help restore full communications in as short a time as possible.
This is especially important in “smoking hole” situations, where infrastructure has been completely devastated, and the company needs to start from scratch, such as the aftermath of the 9/11 attacks. There, AT&T was able to replace a centre that was damaged when the World Trade Centre towers collapsed.
The company’s units and vehicles are all able to interlock and connect to each other, meaning no time is lost in setting up and deployment, with AT&T also able to pre-install and pre-configure trucks to the local environment settings before deployment.
Fortunately, the team has not been called upon for any work in the UK yet (and hopefully will not be) but worked closely with testing before the 2012 Olympics, where it was able to assist with seeing how the Games’ network would cope with the huge amounts of traffic expected.
“Comms is everything,” Williams says, and with deployments lasting anywhere from a few days to six months, it is best to ensure the facility is well-prepared. This means that whenever disaster strikes and the NDR team is needed, the team can be relied on to leap into action quickly and ensure affected areas are back up and active as soon as possible.