Intuit and the Tyranny of the Uptime Clock

Those of you following my Twittering and blog posts must think I have become obsessed with the Intuit outage.  At CSG we operate a hosted enterprise software service and face the tyranny of the uptime clock -- just like Intuit.  As the technology industry moves to adopt cloud computing, we all suffer a credibility loss when a major player like Intuit has a long term outage like this one.  The lack of an explanation, and generally poor levels of communication by Intuit during this episode does not help.  Sure they could not post on their own websites while down, but they have official blogs outside of their control that were up and so they did have the ability to communicate.  Here is a short list of the communications:

Intuit on Twitter @Intuit: First post was 11.5 hours into the outage, at this writing 8 posts, including a gap of 16 hours before the latest post saying they are now on the way back up.   The posts pointed to their community page with 4 undated or time stamped updates, and 2 references to the small business blog, where they posted an update 12 hours into the outage.

Quicken on Twitter @Quicken: First post was 13 hours into the outage, at this writing only 4 posts -- saying they are working on it.

Official Quicken Blog:  No posts, last post was April 26th.

Quickbooks on Twitter @Quickbooks: No posts, last post was May 21st.

The main site just now came back online -- making the outage approximately 34 hours in duration.  Current explanation: 

Our preliminary investigation indicates the outage occurred during a routine maintenance procedure Tuesday night. An accidental power failure during that procedure affected both our primary and backup systems, taking a number of Intuit websites and services offline. While power was quickly restored, we're working diligently to validate our systems and bring them back into full operation.

Intuit reported 300,000 online customers in May of this year -- many of whom use accounting and merchant services applications that require near universal uptime.  In the industry this is often referred to and "four nines" or "five nines" uptime for 99.99% and 99.999% uptime.

A few basics about uptime:  Scheduled outages are usually not included in the calculation, so the .001% downtime permitted in five nines uptime buys only 5.26 minutes of unscheduled downtime in a year.  Three nines gets you almost an hour, and two nines gets you almost a day.  Fortunately nobody died in this outage, so even a 34 hour outage is not a catastrophe on the BP scale.  But it will take 388 years of perfect uptime before Intiut can claim five nines of uptime.

All of us are relieved that they are back online.  This event will undoubtedly slow migrations to the cloud, and should give all of us reason to check and recheck our redundancy and uptime plans.  In addition, we should be checking and rechecking our communication plans associated with any downtime.  We are certainly capable of turning a bad situation worse by failing to communicate well with customers.

Intuit's Cloud Outage

When I saw some traffic in Twitter last night about Intuit's web site going down I first thought it would be back up in a minute and would be no big deal.  An hour later I checked back -- still down.  I checked @intuit, @quicken, and @quickbooks on Twitter thinking they would post an update -- none.  I searched for Intuit related blogs -- no posts in over a month!

Knowing that millions of people use intuit's accounting, payroll, tax preparation, and merchant services, and thinking that these activities are almost always time sensitive, I naturally thought that this was going to be a big story. Next I thought that since small businesses are a big deal for technology companies, and technology companies want small businesses to adopt cloud computing, this would be a big deal in the technology industry.  A major vendor like Intuit going down for hours without any communication to its users is enough to set back cloud adoption a few years -- right?

So I searched the news on Google about a story.  Top search result: Intuit press release about low cost Payment Solutions, next was CEO Brad Smith being profiled by "Inspiring West Virginians", and the next four were all about stock performance upgrades due to good recent financial results.  No stories about the outage.

Back to Twitter, a real time search for Intuit Down:  just a few tweets.  Nothing like I expected.

When a $3 Billion company with millions of customers goes down for over 12 hours without a mention in the press, without an update to its customers, and without any public outcry to speak of I can only conclude one of three things:

1. I fell down the Alice in Wonderland rabbit hole and never came back, or 
2. Not communicating during a crisis works -- no communication = no crisis, or
3. Not that many people are using Intuit's Cloud Services.

I did watch the movie and even with the 3D I don't think I am chasing the Jabberwoky -- so #1 is out, BP and others will tell you that #2 is not true, so logic tells us that #3 could be the most rational conclusion.

Intuit's recent acquisition, Mint, has stayed up the whole time.  No mention on any of Mint's blogs either.