Maintenance error caused Facebook's 6-hour outage, company says


Facebook logo and stock graph are displayed through broken glass in this illustration taken October 4, 2021. REUTERS/Dado Ruvic/Illustration

(Reuters) -An error during routine maintenance on Facebook's network of data centers caused Monday's collapse of its global system for more than six hours, leading to a torrent of problems that delayed the repairs, the company said on Tuesday.

The outage was the largest that Downdetector, a web monitoring firm, said it had ever seen. It blocked access to apps for billions of users of Facebook, Instagram and WhatsApp, further intensifying weeks of scrutiny for the nearly $1 trillion company.

At a U.S. Senate hearing on Tuesday, a former employee turned whistleblower accused Facebook of putting profits before people's safety, which the company denies.

In a blog post, Facebook Vice President of engineering Santosh Janardhan explained the company's engineers issued a command that unintentionally disconnected Facebook data centers from the rest of the world.

Facebook's systems are designed to audit commands to prevent mistakes, but the audit tool had a bug and failed to stop the command that caused the outage, the company said.

The outage was not caused by malicious activity, it added.

While users lost access to one of the world's most popular messaging apps - WhatsApp has more than 2 billion users - employees were also blocked from internal tools.

The outage knocked out tools that engineers would normally use to investigate and repair such outages, making the task even more difficult, Facebook said.

The company said it sent a team of engineers to the location of its data centers to try to debug and restart the systems.

However, it took the company extra time to get engineers inside to work on the servers due to the high physical and system security in place.

Even after network connectivity was restored to the data centers, Facebook said it worried a surge in traffic would cause its websites and apps to crash.

But because the company had run drills to prepare for such situations, access to its services returned relatively quickly.

"Every failure like this is an opportunity to learn and get better," Janardhan wrote. "From here on out, our job is to ... make sure events like this happen as rarely as possible."

(Reporting by Sheila Dang in Dallas; Editing by Sonya Hepinstall, Grant McCool and Richard Pullin)

Article type: metered
User Type: anonymous web
User Status:
Campaign ID: 1
Cxense type: free
User access status: 3
Join our Telegram channel to get our Evening Alerts and breaking news highlights

   

Next In Tech News

Putin warns of "cyber-aggression" against Russia, promises security shakeup
Netflix to pay $59 million to settle Italian tax dispute
Tinder-owner Match says Google to allow for alternate payment systems
Performing artistes push for copyright protection from AI deepfakes
New app urges people to report shark sightings in US
Pakistan TikTok star faces backlash over forest fire video
‘Twitter philanthropy’ reveals chasms in US social safety net
Tesla wants China staff to stay in factory bubble until mid-June
SpaceX paid US$250,000 to settle Elon Musk sexual harassment claim, Insider reports
Computer chip giant ASML places big bets on a tiny future

Others Also Read