Standing before Clear Channel Seattle's new 36,000-square-foot facility on Elliott Avenue, you wouldn't know that inside is an extreme studio network with more than a million crosspoints connected through a 1,232 x 1,232 audio matrix shared between 21 studios and seven stations.
The goal wasn't to build a network this large when the lease ran out on Clear Channel Seattle's studio facility just a half mile up Elliott Avenue, but that's what happens when you cross 77 Wheatstone IP88a Blade access units with 15 Wheatstone control surfaces, 12 crosspoint controllers, three producer turrets, 43 headphone panels, 23 mic control panels and 45 mic processors. And as if that's not enough networking, Clear Channel plans to add at least seven more IP88 Blades, 17 mics, 17 headphone panels and 17 mic processors before the end of the year.
This is one large AoIP network. So, why the big network for seven stations? Why not? And, why don't other stations have networks this large?
It's been more than a year since Clear Channel made the switch, and the benefits are evident at just about every juncture of daily operation. For example, the cluster does a lot of remotes, and like most radio operations today, has a finite complement of remote gear that is always in demand. Sports stations KJR-AM and KHHO-AM are always carrying the latest game, while the music stations -- AC KBKS-FM, Classic Hits KJR-FM, Alternative KKBW-FM, AC KUBE-FM, and Country KNBQ-FM -- are always covering one event or another in and around Seattle. Now, this vital connection to the community is more easily managed and executed due to the system's centralized management of shared resources.
Another benefit of a network of this size and scope -- being able to fully integrate the cluster's RCS NexGen automation system with audio routing for full studio control anywhere on the grid. The stations had been using the automation system since 2001; it is the heart and soul of the operation. Staff schedule music, log in field reports from iPhones and transfer audio files between other Clear Channel stations all over the country using NexGen and an ISDN or Internet connection. Talent can even create an audio segment at a home studio, for example, and drop it into NexGen as an event in the program log.
Tight integration of the AoIP system and NexGen automation was an important consideration. With WheatNet-IP networking, all GPIOs would move off the automation to IP routing access units, which promises transparent interoperability between consoles, automation and networking -- and on down to microphones, recorders and other elements in the studio. Clear Channel could replace soundcards with software drivers where applicable, and replace much of the hardware I/O switching needed for connectivity previously.
- continued on page 2
Before and after
The cluster had been using the SAS 64000 routing system with traditional PR&E consoles before, so IP control surfaces hanging off an AoIP network was an entirely new proposition. Given that they were dealing with a clean slate, management was optimistic that it could take full advantage of the latest routing technology to optimize space and equipment. Working with the engineering staff at Wheatstone, Clear Channel specified a network that could give full remote access and control of devices, automation and surfaces from anywhere on the audio grid. It was an elaborate plan that would let staff start/stop recorders, change levels on sources, bring up audio on a console channel, plus pot up the fader, and then turn the channel off when done. The idea was to implement a complete, intelligent network that could handle everything through one common interface, from audio routing, metering, and leveling to mixing control, logic control and processing adjustments.
To make it happen, all these requirements were rolled into a preconfiguration plan, a 5MB Excel Bible of crosspoints, elements and every input, output, IP address and switching designation in the network.
Putting it on paper was one thing. Getting it implemented was another matter entirely -- especially since Clear Channel had only 30 days to make the switchover due to leasing incentives and penalties that moved up the deadline.
Early on in the project, someone offered the sage advice to take it one day, one studio, and one crosspoint at a time. It was good advice, and this became the refrain throughout the ensuing weeks.
- continued on page 3
Putting it all together
A lot of equipment needed to come together to make the plan happen.
IP88a Blade access points and E-1 and E-6 control surfaces left the Wheatstone factory in New Bern, NC, labeled and preconfigured for installation. Blade IP access units have the Linux operating system built in, with GPIO, all of which needed to hook into the audio grid. Some were set up to handle all digital I/Os, others analog only I/Os, and others were a combination of digital and analog I/Os with mic inputs.
While equipment was en route, studios were networked together through CAT-6 cable. Large as this network is, gigabit edge switches were used to aggregate the Blades in each studio, and each of the 20 switches was then connected to a core switch. Four Cisco 3750G core switches are stacked together in what Cisco calls their Stackwise technology to spread redundancy over four switches. For extra protection, each switch has separate, redundant power supplies.
All furniture was custom made by Studio Technologies, which it shipped to the facility with some light assembly required. As each studio was built out, engineers dropped in control surfaces and hooked up IP88a Blades as needed. Initial testing was done before moving over equipment from the old studio prior to the final switchover.
Clear Channel engineers and contract engineers were brought in from all over the country to help wire the new facility and make the switchover in small steps. Some had never worked with AoIP before, and it was "learn as we go" for the most part. The deadline was pressing, the coordination of details was mind-numbing, and the hours were long. At one point the team was so dogged tired, it took hours to troubleshoot a simple error: Someone had transposed an XLR wire.
When it was all said and done, the facility ended up with more than 77 access points, each with 16 inputs and 16 outputs -- or more than a million crosspoints feeding all 21 studios with soft logic I/Os comprising 4,928 elements and hardware I/O logic of 924 elements. It's a fast operation. One gigabit/second is shuttled throughout, so there are no quality of service (QoS) issues; there's no need to prioritize audio. Fortunately, gigabit Ethernet hardware, which offers point-to-point audio latency of less than two milliseconds, has come down significantly in price in the past five years.
If it's on the grid, it's as good as an arm's reach away anywhere in the facility -- and sometimes outside the facility. If the weekend jock forgets to leave fader no. 6 up for the Saturday evening show, for example, all the engineer has to do is log into the station's network through a laptop and change it with a couple of clicks of the mouse. It's an easy way to handle all sorts of station upsets and operations from home without ruining an afternoon BBQ or letting the beer get warm.
- continued on page 4
It is still an adventure to walk through the 36,000-square-foot facility. Each music station has an on-air control room with new E-6 control surfaces and accompanying producer studio, some with Wheatstone SideBoard mixers for the occasional producer mix-in during a live show. The sports AM, KJR-AM, has a control room with an E-6 console along with producer studio and a talk studio with a SideBoard surrounded by microphones.
In addition, KJR-AM has a Sports Update studio with an E-1 console that can go directly to air, or to the KJR-AM control room as needed. The sports complex also has two sports edit workstations, each with SideBoards and Blade X-Y controllers, so sports updates and other audio can be routed to any one of the control rooms that want to grab them.
A dub studio is for voiceover recording and for general purpose use; it can be switched to air if need be from its E-1 control surface.
Edit 1 and Edit 2 studios are typically used for editing production or voicetracks. Another studio called the Multipurpose Studio has an E-1 console, which is used for production but is also configured as a backup should an on-air studio need to be taken offline for any reason.
Off to the far end of the building is the traffic center made up of 17 cubicles that monitor traffic in Seattle, Portland and other large cities in the region. Each cubicle is outfitted with newsroom mixers, which are networked through IP88a Blade access units to bring up audio from anywhere in the network and to route audio directly to air if needed.
A nearby bullpen for traffic news has controllers so that producers can grab audio feeds from anywhere they want, and there's another IP88a Blade that routes the microphone from these guests into the system so any one of the studios can pull up any one of these workstations to get a traffic report. The bullpen and traffic center are currently being retrofitted to move in the Traffic Network operation, an entity that was recently purchased by Clear Channel.
Four ProTools studios with ProTools consoles round out the studio complex. These studios have all the I/O wired up to IP88a Blade access units to bring audio in and out in the eventuality that Clear Channel retires the ProTools consoles and puts in new IP control surfaces.
- continued on page 5
Along the hallway in the center of the studio complex and separate from the rack room is what is called the Remote Broadcast Rack. All ISDN, POTS codecs, Marti RPUs, and any other remote equipment is located here for shared access by staff and stations. Staff can select any of the gear from their E-1 or E-6 consoles anywhere in the studio, allowing them to share existing gear rather than Clear Channel having to buy new equipment. Instead of buying new ISDN units, for example, the cluster has six ISDN units that staff from any of the seven stations can use on a rotating basis.
The physical and operational changes that come with an IP operation this size are significant. For starters, the control surfaces are a huge departure from the older consoles, which you couldn't possibly manage without moving from one end of the board to the other. Now, operators can stand in front of the E-1 or -6 and have full range of control -- controls that can be changed at any time by different board operators.
There's obviously a lot going on under the hood, too. It's a very dynamic environment. Devices are always talking to each other about command and control, and not just streaming audio between them. This gives the studios unbelievable flexibility. In fact, no sooner had the team finished installation when they began making changes. Two studios have changed function entirely, and currently, Clear Channel is in the process of converting another multipurpose studio into a dedicated studio for a syndicated country show.
Once the networking was laid out, though, the rest was easy because it's just a matter of changing crosspoints around and relabeling a few connections. Now changes are made as a matter of routine, which is pretty remarkable considering the size and depth of this networked operation.
Cisco 3750G Stackwise switches
Comrex Access, Matrix
Fostex CR500, RM-1
RCS NexGen automation
Studio Technologies furniture
Telos Zephyr Xstream, 2101
Wheatstone E-1 control surfaces, E-6 control surfaces, IP88a Blades, SideBoard mixers
■ Erik Kuhlmann, senior vice president of engineering, Clear Channel Media + Entertainment
■ Dan Mettler, senior vice president of engineering, Central Region, Clear Channel Media + Entertainment
■ Steve George, regional vice president of engineering, Plains Region, Clear Channel Media + Entertainment
■ Bill Major, assistant director of engineering, Seattle Clear Channel Media + Entertainment
■ Terry Ryan, remote coordinator/engineer, Seattle Clear Channel Media + Entertainment
■ And the many other Clear Channel engineers who came to aid in the installation. Their help was greatly appreciated.
Major is assistant director of engineering, Clear Channel Seattle.