What size pallete are you talking about? Is this fullcolor or grayscale?
Also, when you say “closest”, what color distance metric? If you want the closest in RGB colorspace then you can compute that efficiently with
d = (dx * dx + dy * dy + dz * dz); // no sqrt required
as you are basically sorting – for related discussion see https://stackoverflow.com/questions/3693514/very-fast-3d-distance-check
Note however that RGB distance is really rough – in some places it won’t mimic intuitive perceptual colorspaces, as it is based output – not the eye.