For AM you'll need a device to switch the transmitter input between the two evaluation devices. This could be as simple as a DPDT switch (for analog audio) actuated by a remote control. Make sure there is some sort of status that tells you definitively which device is actually on-air.
For FM, obtain a composite switch, or perhaps build one yourself (SPDT coaxial relay). Again, make sure there is status that allows you to know which one is on-air.
Clearly both test devices should be fed by exactly the same audio source. Peak modulation levels generated by the two devices need to match for a fair test. An oscilloscope is the best instrument for setting peak levels; make sure the evaluation device is actually connected up to its load when setting the levels. (We are striving for accuracy, after all.)
After the setup is done comes the real work. You will need to spend time adjusting device A to your liking, and you'll need to spend time adjusting device B to your liking. (Quite often one of the two devices will be the legacy unit so most likely it'll be tweaked-out already.)
There are two primary locations for evaluating the devices: in your vehicle, and on a monaural radio in an office environment. Your vehicle is important not only because of the in-car listening aspect that is so important to radio in general, but also because it's probably the source of sound you are most familiar with. That familiarity is important because it makes differences in sound from one device to the other easier to pick out. An office environment is also important in the grand scheme of radio; but more specifically, you'll want to hear the station in mono like so many listeners will. Radios such as these often don't have a lot of bass response, and they don't have a lot of high-frequency response, so you'll probably find the station sounds different on one of these versus a system that has three-way speakers, and good wideband audio response. (The vast majority of listeners don't hear your station that way.)
When running comparative listening tests between devices, it's important that, at the very least, you hear all the various sources that get sent out. For an FM music station, that might take a couple of hours, or a couple of days depending upon the format. For a talk station, make sure during the A vs. B evaluation period, you hear all the various voices and production sources.
This is an iterative process; during this A vs. B period, you may decide to do extra tweaking on one or both devices. Use your remote control to switch processors at any time; during speech, during songs, during spots. Listen to the differences between the devices on all sources and at different times (i.e., different jocks or board operators). This is also a great way to compare your stations' potential new sound to any competitors. I would also caution you not to jump to conclusions too early on. Give it a couple of days at minimum - listen a lot - and after that amount of time it's likely the winner will have emerged.
As I said before, the decision on what to do with on-air processing should be subjective after you've determined what the practical choices are. There isn't any real point in evaluating the on-air sound that you can get from a device that, ultimately, the station can't afford.
Irwin is transmission systems supervisor for Clear Channel NYC and chief engineer of WKTU, New York. Contact him at firstname.lastname@example.org.