Got it! Are you planning to have a straight-overhead camera, or should it be robust from multiple angles?
In either case, you might want to consider camera calibration – and/or marking / registering your corners. For example, with BoofCV or OpenCV libraries
https://boofcv.org/index.php?title=Tutorial_Camera_Calibration
Yellow is suffering from glare. However, if you have a calibrated grid then you can just detect which color pixels fall in each grid square, and make each whole square equal to the most occurring color.