1
$\begingroup$

I want to predict how open is the mouth given a face image. It's a regression problem (0= mouth not open, 1=mouth completely open). And something between 0 and 1 is also allowed. ConvNet works fine for one person. But when I train with many people with hope that it will generalize to an unseen person, the model suffers from not knowing the limit of a person's mouth.

For example, if a new person uses the model to predict, the model doesn't have a clue whether this person has completely opened the mouth or not. Because it's hard to know how much a person can open the mouth from one image. People's mouth openness capability is not the same. Some guys cannot open their mouth that much, but some guys can open the mouth like they can swallow an apple. The only way you can know how much a person can open the mouth is to look at multiple images of their mouth movements, especially when they open the mouth completely.

I want to know how to make the model know the limit of a person's mouth by using the info from past images.

Is there a way for me to use a few unlabeled images of a new person in order to help the model calibrate its prediction? How do I do it?

This should help the model know the min/max of the person's mouth and also knows the intermediate values between 0 and 1. If you run the model continuously on a webcam, I expect the prediction to be smooth (not noisy).

My idea is to encode many images into an embedding that can be used as a calibration vector. The vector will be fed into the model along with the person's image. But I am not sure how to do it. Any suggestions/tutorials would be welcomed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.