There are many tools and libraries focused on counting groups of colors in an image. In general, counting many colors as one color is called “color quantization”:
Related past discussion:
If you want to group colors using a strategy that maintains visual qualities of the original color space, perhaps start with Median Cut – we had a past conversation pointing to the Java implementation of Median Cut available from ImageJ. javax also has a ColorQuantizer.
OpenCV also has some quantization built-in via k-means clustering. There is a good discussion of approaches using OpenCV here:
I’m guessing that is what you mean – rather than using a stock palette, you want to adaptively choose a small number of colors that best approximate the appearance of your original image. Is that right?