You may want to try using the opencv library to get a more robust background subtraction algorithm. It uses an “adaptive” algorithm – although depending on your goals you might want to drop the entire background subtraction thing and go with learning and object detection.
At any rate, the algorithm and its paper is described here:
https://kyamagu.github.io/mexopencv/matlab/BackgroundSubtractorMOG.html
…and it is built into the opencv processing library, an older example of a background subtraction sketch:
I haven’t read it, but there might be a more updated discussion / example in this recent chapter:
Chung B.W. (2017) “Understanding Motion.” In: Pro Processing for Images and Computer Vision with OpenCV. Apress, Berkeley, CA