The Digital Face-Plant: Thoughts on IP Security

June 26, 2013

Few things matter as much in classical music broadcast engineering as audio fidelity. That's one of the reasons The Classical Network plunged forward a few years ago into the world of digital radio. Regardless of what digiphobes tell you, analog technology and analog FM cannot rival digital audio and digital radio (HD Radio) for music fidelity in at least four areas critical to musical genres like classical music: frequency response, signal-to-noise ratio, dynamic range, and eliminating musicality-killers like limiter clipping.

But, as we have learned the hard way at The Classical Network, digital paths offer the opportunity to plant your teeth firmly in the gravel. The painful experience comes with a caveat emptor: the folks who sold you the digital technology can disappear as quickly as data packets when the problems outrun the solutions.

Three years after abandoning satellite delivery for our network stations and entering the brave new world of Internet delivery to get better audio quality, we are returning to satellite audio delivery to avoid hackers and restore consistent audio. In the process, we are accepting lower quality audio in exchange for the even-more-important coin of the realm in broadcasting: reliability. We learned valuable lessons by being early adopters, but some of the mistakes could have been avoided.

Ironically, our odyssey began as our audio improved, not as we searched for better audio. As our flagship station's audio technology grew steadily better over the past decade, we noticed that our satellite audio -- stable, but 30-year-old technology -- didn't sound all that stellar in the field. The ubiquitous stringed instruments of classical music often sounded more brittle and ragged than they did in the day of tube transmitters and telco-loop studio links. Improved technology feeding the satellite uplink and following the satellite downlink made other audio problems in the chain -- including the satellite chain -- even more obvious.

About the same time these audio issues became more apparent, we launched our HD2 network of jazz stations. Faced with a need to double our satellite bandwidth (meaning higher costs for two channels of marginal audio), we started looking at our options.

One of our more-trusted equipment vendors suggested that we look at IP codecs that he assured us were popular in Europe. The codecs promised audio that was nearly indistinguishable from uncompressed digital audio, even when delivered over public Internet.

When we heard the codecs, they did sound impressive. Some of our hosts with highly trained ears could hear subtle differences between the original uncompressed audio and the codec algorithms, but the codecs were vastly superior to the audio we were getting from our satellite system. Better still, with some careful failover configurations coupling low-cost bandwidth from two or three business-class Internet providers, we calculated that we could run both our classical and jazz networks over public Internet for less than we were paying for just mediocre classical music delivery by satellite.

After we launched, the system garnered us some trade publication ink, and it worked so well that we quickly started buying the blue rack units for more locations and adding more bandwidth. We added the units to venues in New York City and Philadelphia. Eventually, we were broadcasting live concerts that provided a fully-digital experience for classical music lovers anywhere in the world in stunning quality, just a few seconds after the same music was heard by those sitting in the famed venues of New York and Philadelphia.

We succeeded in looking toward the future of classical broadcasting: we delivered fully-digital audio from NYC to our studio, then out our digital radio pathway to NYC, Philly, and New Jersey, plus to Internet listeners around the world, all within seconds of the live performance. It was all kept in the digital domain, in stunning quality.

- continued on page 2

Unfortunately, the cutting edge can quickly become the bleeding edge. Configuration of the codecs was not intuitive, and models and features appeared and disappeared far more quickly than the manufacturer's technical assistance could cover. Maybe it was inevitable that as quickly as it took off, two headline-grabbing events brought our expanding system to its knees.

First, Superstorm Sandy clobbered the Northeast, and the nation's most populated region took a noticeable hit to its Internet infrastructure. For several months after Sandy, our business-class Internet providers could not consistently provide the bandwidth we normally counted on from them. We could go from stable operation to nothing within seconds, then back to stable within hours...or days. The brave new world of public Internet-based audio proved remarkably fragile.

Even half-a-year later, reliability still fluctuates as major Internet providers work to replace damaged fiber and old copper in places like lower Manhattan, and our attempts at finding wireless bandwidth out of our partner venues there have proven fruitless because of Manhattan's famous canyons of tall buildings.

The other news headline that unexpectedly crossed our path was the increase in Internet hacking and denial of service type attacks. Our first inkling of how hackers could hurt us came shortly after installing a fully digital transmitter at one of our stations along the Jersey Shore just weeks before Sandy hit. Very shortly, our major Internet provider there notified us (with less than five minutes warning) they were pulling the plug on our broadband connection because their security department had detected a spam phishing operation emanating from our transmitter building.

Not coincidentally, our operation sat inside the Internet provider's compound, literally hardwired to the head-end of their massive broadband fiber network. The hackers had a huge broadband pipe at their disposal, and a Linux-base control system in our new transmitter to act as their spam server. Life was good...for them.

The vulnerabilities in that particular transmitter had been well publicized in the hacker community (somehow, we missed reading it!), with all the hacking information readily available on the Internet. Unfortunately, without any safeguards or warning from the transmitter's manufacturer, we were just one more sitting duck.

That brief and blunt primer taught us two things about how the digital age has changed broadcasting and the requirements for broadcast engineering. Broadcast manufacturers need to take digital security seriously and practice transparency with their clients about known problems, and IT coursework -- including system security -- is a must for broadcast engineers.

Since our transmitter problem, we have found our IP codec system is as hacker-vulnerable as our transmission systems. The first generation of the codec units -- and these legacy units account for the majority of the units in our system -- have no real security. Their readily available software control system allows anybody who knows the static IP on your codec to easily access it, no password required. In fact, no password protection is available.

It's a throwback to the days of the dial-up remote control system where knowing the right phone number and punching the default password (1-2-3-4-5) into the keypad would get you into many - if not most - transmitter remote control systems in the country. The only differences are that it's easier for hackers to scan for static IPs than it is to find transmitter telemetry system phone numbers, and the dial-up remote controls at least had password protection that could be changed.

Coupled with an already damaged Internet infrastructure, the hacking problem has made our IP system too risky to keep and too costly to update. Dedicated loops quickly make the extensive system more costly than satellite. Sending IT security experts around our network to harden each site is too costly for a public station that must go through public bid procedures before it can cough twice.

So, we're headed back to satellite for our main network distribution. Since we left the geo-stationary orbit guys, their technology has advanced considerably. We can get reasonably good audio for both our classical and jazz networks in the same bandwidth we once used for just mediocre audio for one network. For the time being, satellite audio appears to be beyond the reach of all but state-sponsored hacking operations. When state-sponsored hackers finally manage to seize control of satellite systems, I can only hope the satellite vendor will be more forthcoming with information and solutions than our transmitter and codec vendors.

Meanwhile, we have more than a dozen high-quality, low-security audio codecs that we will use in situations where we can control problems quickly if they occur. Concert venues are where they will serve until they die. In three-hour stints, the known problems are manageable, and they do sound great.

Peter Fretwell is general manager of The Classical Network, based in New Jersey. He is co-author of Lessons from the Hanoi Hilton: Six Characteristics of High-Performance Teams published by Naval Institute Press.

Receive regular news and technology updates. Sign up for our free newsletter here.