Which face is which? Persistence faces tracking

I have this p5js sketch that uses tensorflow Blazeface to track faces.

I was wondering how I would go about keeping track of multiple faces for an interactive installation I am doing?
Kinda like Face-It/FaceDetectMemory.pde at master · shiffman/Face-It · GitHub but perhaps even simpler/more roboust and in JavaScript?

I am not looking for actual face recognition btw.