Color detect with Kinect v2

I detect the color of the glove and check if it is one of the three colors (RGB). Every frame I get integer values:

NColor = 0 for RED
NColor = 1 for GREEN
NColor = 2 for BLUE
NColor = -1 for unknown color

As the lighting conditions are not always ideal, the integer values change frequently. What would be the best procedure to find the frequency of occurrence of a number in a given interval that would more accurately represent the color of the glove?

Any help will be appreciated…