Guest Post by NCX Group Contributor Anthony M. Freed
In an increasingly outsourced marketplace, third-party service providers offer companies multiple benefits, including marked savings on hardware outlays and support personnel, scalability options and the utilization of cutting edge technologies.
With these advantages also comes an element of risk, as clients are potentially subject to costly interruptions and retain little in the way of control when operations cease due to service provider failures.
The lack of proactive business continuity planning by clients of third-party services could result in significant revenue losses, a tarnishing of the enterprise brand, and potentially long term effects in the way of customer confidence.
That’s the predicament hundreds of businesses suffered with the recent service outage experienced by website hosting company GoDaddy.
On Monday of last week around 1pm EST, GoDaddy’s networks began experiencing intermittent service outages which in turn caused a flood of customer complaints conveyed by way Twitter direct messages.
With their networks down, GoDaddy had few options but to keep customers updated on the event through the popular social media platform, issuing messages such as “Status Alert: Hey, all. We’re aware of the trouble people are having with our site. We’re working on it,” and “Sorry to hear all your frustration. We’re working feverishly to resolve as soon as possible.”
To further complicate matters, rumors quickly began circulating that the hosting company may have fallen victim to an attack by a member of the Anonymous movement after a hacktivist who goes by the handle AnonymousOwn3r tweeted several messages in an effort to claim responsibility for the outage.
The possibility that hactivists were behind the service interruption was cause for concern by GoDaddy customers who feared the possibility that sensitive proprietary data stored on the networks may have been compromised.
Services were restored to a functional level by 7pm EST Monday evening, but the harm was already inflicted. In a bad versus worse case exercise in damage control, GoDaddy worked to dispel reports that the event was caused by an intrusion event or denial of service attack.
Sheepishly, in an effort to assure the security of client data, the company issued statements that the service interruption was actually due to improper systems configurations which resulted in corrupted router data tables and the subsequent outage.
“The service outage was not caused by external influences. It was not a “hack” and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. At no time was any customer data at risk or were any of our systems compromised,” CEO Scott Wagner said in a statement.
While the statements may have been beneficial in putting the minds of customers at ease in regards to the potential for data exposure, the admission of errors on the company’s part was less than optimal for a business which counts on guaranteed uptime as a saleable asset.
“We have implemented measures to prevent this from occurring again… Throughout our history, we have provided 99.999% uptime in our DNS infrastructure. This is the level our customers expect from us and the level we expect of ourselves. We have let our customers down and we know it. We take our business and our customers’ businesses very seriously. We apologize to our customers for these events and thank them for their patience,” Wagner acknowledged.
Sure, patience and understanding are virtuous, but there is little doubt that many of GoDaddy’s customers who were impacted by the service interruption are now scrambling to ensure business continuity procedures are up to date and effective should another incident occur.
The lesson for management in companies that engage third-party service providers is one of careful preparation. Companies need to conduct thorough analysis of the processes and resources that are critical to continued business operations in the face of circumstances that may otherwise be beyond their locus of control.
Included in the analysis should be an examination of all vendor-supplied services, service level agreements, comprehensive incident response protocols, and proactive mitigation planning to ensure continuity in the event any of the links in the operations chain are broken.
Simply telling a customer that you cannot make good on contracted deliverables or that access to a public facing portal is not available due to circumstances beyond your control will not suffice, and a third-party incident like that which GoDaddy experienced could have long lasting repercussions on your company’s bottom line.
Anthony M. Freed is a freelance security journalist and editor, and has authored numerous feature articles, interviews and investigative reports which have been sourced and cited by dozens of major media outlets, including The New York Times, Reuters, The Register, Financial Times of London, MSNBC, Fox News, PC/IT/Computer/Tech World, eWeek, SC Magazine, CSO Magazine, Federal News Radio, The Herald-Tribune, Naked Security, and many more.
Photo Courtesy of ☺ Lee J Haywood