Thanks for all the responses! Yes, the output of the device is working into a properly terminated 75 ohm load (input of a 8 x 4 crosspoint switch). No setting of the various menu items provides any real improvement in the waveform, as far as the ratio between sync and video signal is concerned. The 2 DIP switches are used to select PAL or NTSC output, and are properly set for NTSC. The signal is shown on a dedicated NTSC waveform monitor, (Tektronix 1730), not a general purpose scope. Will rig up a terminating plug for the unused VGA passthrough to see if that helps at all.
Neither of the 2 chips you suggested are the one I have here, but that "jammaboard" unit looks VERY similar as far as circuitry if not layout/packaging. The chip on the pictured unit is a VX1937, manufactured by VXIS, and searching for a datasheet shows a chip with the right package and number of pins, which requires the same 27 MHz timebase crystal as in my unit. Looks like we have a match!
Unfortunately, the only datasheet I can find for it is a 2 page summary, lacking any actual data like pinouts, etc. that would be needed for reverse engineering the thing. This is a start, though.