0
$\begingroup$

I'm working on a semantic segmentation project, and want to add some domain knowledge to the system. I want to ensure that for segmentation, there can only be one group of pixels that are predicted as a certain class. An example would be segmentation of the face. When we want to segment different parts of the face, assuming regular faces, we have the following domain knowledge: one group of pixels can be predicted as the mouth; one group can be predicted as the nose; two groups of pixels can be predicted as the eyes.

A possible solution I currently think of is as follows: Suppose we find two groups of pixels, both groups having pixels with the highest probability for being the nose. Then, we would take the average score for each pixel group to obtain two average scores. Then we compute the probabilities again and select the group with the highest probability to be the nose. The group of pixels having the lower probability will then be predicted to be the class having the second best probability. Doing it in this way, we apply some post-processing (look at scores and determine what is our final prediction, using some logic). However, it does not seem to be learned by the neural network (or maybe it is learned implicitly, by looking at the final predicted class obtained from the postprocessing and updating the weights accordingly).

Are there any solutions, or could you point me to any papers which describe this problem and add domain knowledge to the neural network such that it incorporates this information and becomes better at predicting?

$\endgroup$

1 Answer 1

0
$\begingroup$

You could come up with a loss function which describes your constraints.

PS: your example with the face recognition (where a CNN would find eyes no matter how nonsensical they are placed) is probably solved by visual transformers

$\endgroup$
1
  • $\begingroup$ Thank you very much for your answer. Do you also have an example of the loss function? I currently think of counting the number of groups found and applying a penalty if the number of groups exceeds the allowed number. However, is this the most viable strategy? Further, regarding visual transformers, they are very interesting. However, one of the constraints is also that we should be able to run inference in real time. From what I know, ViTs are currently not capable of doing that (yet). $\endgroup$
    – Mark-M2L
    Sep 8, 2022 at 10:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .