How to optimize HSB color comparisons?

This sketch takes a picture and interprets it as though it was woven from colored string wrapped around a circle of nails. It was originally written in RGB space where I am most comfortable. I tried swapping out the RGB comparisons for HSI like so:

// the square of the linear distance between two colors.
float scoreColors(color c0,color c1) {
  float h = hue(c0)-hue(c1);
  float s = saturation(c0)-saturation(c1);
  float b = brightness(c0)-brightness(c1);
  return (h*h +s*s +b*b);
}

and it’s great! but also slow AF. I’m looking for ideas how to reduce the total number of repetitive HSB conversions. One I had was to somehow convert the entire source image via some HSBImage class and then work entirely in HSB - except when writing out to the visible buffers. Already it sounds like a mess.
I tried using colorMode(HSB) but that’s not helpful, colors were wildly wrong and it didn’t seem to run any faster.

So if you have some ideas I’d love to hear them. Low priority, just a hobby for me.

Thanks!

oh and here’s some examples. with RGB:

with HSB: