Managing Technology: Thinking About Disaster Recovery; Response-Based Planning

February 1, 2015

Bad stuff happens, right? Even in a well-built and maintained operation, there’s always something that threatens to take you off-air: bad weather, power utility issues, human error (“oops”!). Even something like your air conditioner breaking late on a Friday night can temporarily put you out of business. Those and many other incidents all fit under the “Disaster Recovery” heading.

Why worry about disaster recovery? The simplest and cheapest thing to do during a threatening incident is to shut everything off, hunker down and wait for the bad situation to pass. What’s wrong with that? Preparing for incidents isn’t an abstraction or a luxury; you need to respond when there’s a failure.

If you aren’t prepared to respond to an incident (or don’t have a good plan to exercise those preparations), you’ll lose money, audience and respect. Simple as that. If you’re off the air, you’re not making money. You can do the math to calculate how long you can be offline before you’ve eaten up any “savings” from not being prepared. There’s a secondary cost, too. If you’re off the air when your audience needs your services, they’ll go to someone else and maybe not come back. There’s also the commitment radio stations have with their communities to provide useful information during incidents; if you’re off the air, you’re not meeting your commitments.

OK, you have good disaster recovery preparations and lots of redundancy in your plant. You’re ready to go, right? If you have disaster preparations but don’t have a written, tested, practiced and curated plan, your station isn’t ready for an incident. Broadcast facilities can be fairly complex operations, especially when you factor in translators, HD Radio channels and online streams, all with different (or common) content stores and technology.

Do any of the following sound familiar?

•   “I’ve been meaning to write something down, but it’s just been too busy … forever.”

•   “Yes, we have a disaster recovery plan; I keep it in my head — so all anyone has to do is call.”

•   “That was an intern project five or six years ago.”

•   “Yes, we had one, but it was impossible to maintain. There are just too many ways things can go wrong. 

"If you aren’t prepared to respond to an incident (or don’t have a good plan to exercise those preparations), you’ll lose money, audience and respect. Simple as that. If you’re off the air, you’re not making money."

A lot of disaster recovery plans are unwieldy, burdened by a seemingly infinite number of specific scenarios. Some, good intentions aside, never get written down because of the daunting amount of detail required. They’re written from a troubleshooting perspective instead of an operations perspective. The plans are written by engineers who want to understand what’s gone wrong rather than taking protective action to keep their service alive. In other words, a lot of disaster recovery plans are cause-focused rather than response-focused.

Cause-focused scenarios aren’t an inherently terrible way to think about disaster recovery. It helps to think through a specific incident as a template for the preparations needed to respond and recover. A cause-focused scenario could be “the air conditioning died,” which generates procedures on troubleshooting and fixing the issue. Additional procedures describe what operations and on-air staff needs to do — in this case, get ready to shut down unneeded devices or get ready to move to a backup site.

As you add more scenarios, you get a detailed list of possible causes of service interruptions with overlapping response plans. So far, so good.

Receive regular news and technology updates. Sign up for our free newsletter here.

Comments