# Keeping track of multiple faces throughout a video

I have a video where multiple persons are seated. I need to keep track of the emotions they show throughout the video. My final result should be a csv file with all the emotions depicted by each person for every 1 minute.

So it would look like this :

Person 1 : Angry Happy Sad ....


I used RetinaFace to detect all the faces present in selected frames from the video.

It worked well for the frames detecting almost all the faces. However I would like to keep track of the faces detected throughout the video, as I want to track the emotion depicted by each person throughout the video. But since the locations of faces change throughout the video, I'm a little confused as to how I can accomplish this.

Some Ideas :

These are the annotation files for the first 2 frames used :

0,"[1811, 850, 1948, 1013]",0.999666452407836,"[[1828, 911], [1887, 913], [1841, 942], [1832, 974], [1876, 976]]"
1,"[346, 1285, 503, 1468]",0.9996420145034791,"[[365, 1361], [424, 1348], [385, 1395], [390, 1426], [439, 1416]]"
2,"[1543, 1418, 1702, 1618]",0.9995224475860591,"[[1578, 1514], [1647, 1498], [1619, 1554], [1610, 1585], [1658, 1572]]"
3,"[1191, 838, 1323, 1006]",0.9994902610778801,"[[1215, 889], [1273, 898], [1228, 930], [1212, 959], [1257, 966]]"
4,"[2153, 857, 2293, 1047]",0.9993738532066341,"[[2170, 928], [2227, 925], [2182, 956], [2175, 996], [2219, 992]]"
5,"[303, 860, 423, 1031]",0.9985873699188231,"[[315, 929], [354, 922], [319, 962], [327, 995], [355, 988]]"
6,"[1596, 356, 1691, 467]",0.998558819293975,"[[1617, 398], [1659, 397], [1633, 418], [1621, 438], [1654, 438]]"

0,"[391, 1246, 541, 1430]",0.9996455907821651,"[[409, 1316], [469, 1311], [423, 1349], [419, 1383], [466, 1379]]"
1,"[1556, 1377, 1708, 1577]",0.9996392726898191,"[[1583, 1466], [1650, 1455], [1610, 1491], [1597, 1533], [1648, 1526]]"
2,"[1806, 862, 1940, 1031]",0.999588668346405,"[[1827, 928], [1887, 926], [1848, 963], [1836, 992], [1884, 990]]"
3,"[678, 876, 812, 1063]",0.99949049949646,"[[691, 954], [743, 946], [707, 989], [712, 1019], [751, 1011]]"
4,"[2192, 873, 2338, 1047]",0.999396681785583,"[[2213, 950], [2260, 936], [2232, 983], [2244, 1010], [2283, 998]]"


From these files, we can see that [1543, 1418, 1702, 1618] and [1556, 1377, 1708, 1577], [346, 1285, 503, 1468] and [391, 1246, 541, 1430] are similar to each other, so they might be the same faces. Also throughout the video, all the persons will be seated, so their positions might not differ much. But I do not know what margin I could use to find this.

I would like to know if there are models for keeping track of the faces throughout the video.