You can do it on binary buffers for each channel if you do it after quantization. During nearest-pixel, the results of each nearest-color channel is written to a different buffer, which can be processed as a binary image (eg erode dialate). The buffers are then combined.