If you use HSB instead of RGB, wouldn’t be the “distance” more perceptually correct?
(NB- I don’t reallly believe in the term “perceptually correct”. I am colorblind and know non-colorblind people suffer an exaggerated perception of some dull color they call “red.”)