Ah - I think I see the problem. You did the filtering for the source signal as some sort of Photoshop motion blur, correct? It is showing very sharp transitions into and out of the "blurred" area between the magenta and green. This is not normal filtering, and results in strange effects.
Here are some simulation results with proper filters in Photoshop. The simulation includes the modulation and demodulation, so cross-color (orange/cyan moire' in striped awning) and cross-luma (dots on vertical edges of the bars) are visible.
The smaller +/-I and +/-Q bars at upper left center have no Y variation, so you are seeing just the effects of I and Q there.
There is some anamorphic stretch due to adjustment to get the color subcarrier dots approximately correct coarseness.
Original:
veg market with IQ bars and cbars by
old_tv_nut, on Flickr
chroma on gray with transmitter I and Q filters:
chroma in IQ filtered by
old_tv_nut, on Flickr
Demodulated chroma on gray with both transmitter and receiver I and Q filtering:
chroma out IQ code + IQ decode by
old_tv_nut, on Flickr
Demodulated I:
I out as mono by
old_tv_nut, on Flickr
Demodulated Q:
Q out as mono by
old_tv_nut, on Flickr
Final output:
final out IQ code + IQ decode by
old_tv_nut, on Flickr
Both the cross color and cross luma are more visible than normal because they are stationary.