Categories: Cloud

CloudFlare Gets Caught Out By 2016 Leap Second

The extra leap second added on to the end of 2016 may not have had an effect on most people, but it did catch out a few web companies who failed to factor it in.

Web services and security firm CloudFlare was one such example. A small number of its servers went down at midnight UTC on New Year’s Day due to an error in its RRDNS software, a domain name service (DNS) proxy that was written to help scale CloudFlare’s DNS infrastructure, which limited web access for some of its customers.

As CloudFlare explained in a blog post, a number went negative in the software when it should of been zero, causing RRDNS to “panic” and affect the DNS resolutions to some websites.

Time warp

The problem only affected “a small number” of CloudFlare customers using CNAME DNS records, with “approximately 0.2 percent” of DNS queries being affected and errors only occurring in less than one percent of all HTTP requests to Cloudflare.

The issue was confirmed by the company’s engineers at 00:34 UTC on New Year’s Day and the fix – which involved patching the clock source to ensure it normalises if time ever skips backwards – was rolled out to the majority of the affected data centres by 02:50 UTC.

CloudFlare’s CTO John Graham-Cumming said: “This problem was quickly identified. The most affected machines were patched in 90 minutes and the fix was rolled out worldwide by 0645 UTC.

“We are sorry that our customers were affected by this bug and are inspecting all our code to ensure that there are no other leap second sensitive uses of time intervals.”

This is not the first time a leap second has been an issue. Back in 2012 an extra second caused technical problems for the likes of LinkedIn, FourSquare and Reddit, resulting in IT organisations bracing themselves for panic ahead of a leap second addition in 2015.

Quiz: Test your web browser knowledge

Sam Pudwell

Sam Pudwell joined Silicon UK as a reporter in December 2016. As well as being the resident Cloud aficionado, he covers areas such as cyber security, government IT and sports technology, with the aim of going to as many events as possible.

Recent Posts

Regulator Demands Answers Over Tesla FSD Social Media Posts

US regulator says Tesla's messaging on social media could lead drivers to believe 'Full Self-Driving'…

2 hours ago

US Takes Chip Equipment Makers To Task Over China Sales

US lawmakers write to biggest chip equpment makers in US, Japan, Netherlands asking for data…

3 hours ago

Nvidia Market Value Surges Above $3.6tn

Investor optimism following Donald Trump re-election win pushes Nvidia to record high, adds 30 percent…

3 hours ago

ASML IT Outage Disrupts Worldwide Operations

Dutch chip equipment maker ASML says 'fully recovered' from IT outage affecting operations worldwide and…

4 hours ago

Judge Dismisses Personal Claims Against Meta’s Zuckerberg

US federal judge finds Meta chief excutive Zuckerberg can't be held personally liable in child-addiction…

4 hours ago

China’s SMIC Benefits From AI Chip Boom In Spite Of Sanctions

Chinese chip manufacturing giant SMIC says AI has helped boost revenues to record levels in…

5 hours ago