Thanks for sharing your solution!
Out of curiosity, are you swapping these colors once, to a cached image, or are you swapping the pixel values every single frame? If performance is an issue, not recalculating the same thing every single frame is another potential strategy.