Risk Aversion and Lessons Learned

Sutro baths in SF(1)I grew up in the Midwest, tornado country and we always had a plan, at school we trained by getting in the hall way, moving away from windows and making ourselves as small as possible.  At home we knew where the candles, and emergency supplies were kept and the safest room in the house.  Living in the Midwest we thought the scariest place in the US to live in terms of natural disasters was in California with the earthquakes and fires.  Now that I live in California, I find it ironic that most people here seem to find the tornados of the Midwest and the hurricanes of the East Coast more frightening.  Regardless of the risk, all of these crisis have one thing in common, when they happen they can be catastrophic, but good preparation goes a long way to improving the outcome.  One of the first things I did when I moved to San Francisco was stock up on supplies, prepare emergency kits, and gather all the information we’d need but might not be able to find.  While not perfect, I sleep easier at night, with my shoes and flashlight by my bed.

I will never forget early in my career working on a fiber network build project in Iowa.  That summer we experienced torrential rains and flooding was an issue throughout the state.  Our warehouse location was located about a quarter mile from a river and knowing the potential of flooding, we had made arrangements with a local freight company to be on call should the river start to crest.  It started rising, we called the truckers and our $10M in inventory was moved to safety, and not a moment too soon.  I had someone on lookout watching the river so that he could come running if it spilled over the sides.  Way too close in my opinion, but we had made arrangements that we’d move the trucks with whatever could salvage, but we had a plan, it may not have been the best but we had the arrangements in place so that with one call we had the help we needed.  Most small companies, and dare I suggest larger ones, do not have any sort of contingency plans in place.  They do not want to consider what could happen.  The problem is that it does not have to be those catastrophic events of a hurricane or an earth quake to bring a company to its knees but a fire in the building, the death of a principal, a stolen computer.

Working on the physical plant side of telecommunications, I constantly had to consider the “what ifs” as our client side contracts guaranteed redundancy with minimum service  disruptions.   To guarantee  that level of service we had equipment and replacement parts strategically placed around the network, and retainer contracts for services that might be required.  Thinking through the what-ifs is a powerful exercise and can even lead to ways of improved processes going forward.

The IT industry just had a wake up call a few weeks back in Seattle when a fire took down several sites, and left a lot of them with egg on their face.  One CTO learned his lesson from a similar incident the precious year, and had a disater recovery plan ready to be implemented that included having redundant systems in place.  The differences were telling, here is a PR message from a company that was not prepared:

Bing Travel:  ”A fire occurred at Fisher Plaza in downtown Seattle just after midnight on Friday morning. The blown transformer knocked out power to the entire building, which is home to the Bing Travel servers.  We’re hard at work to restore service following this unexpected event.”

With proper planning, this would never be an “unexpected event.”

For the piece of mind achieved by the planning, and knowing that the majority of the competition has not done contingency planning making you look really, really good.

  • Share/Bookmark

Leave a comment

This site is protected by WP-CopyRightPro