Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar items. Suppose we also have access to embeddings for each item.
A naive approach might be to concat the two item embeddings, add a linear layer or two and finally perform a sigmoid (as this is binary classification) for the output probability.
However, that approach would mean that potentially inputing
(x, y) to the model could give a different score from inputing
(y, x) into it, since concat is not symmetric.
How can we go about overcoming this? What is the common practice in this situation?
So far I have thought about:
Whenever I input
(x, y)I can also input
(y, x)and always take the average prediction of both of them. But this feels like a hacky way of forcing the network to be symmetric, it doesn't make it learn the same thing despite of the input order.
Replacing concat with some other symmetric tensor operation. But what operation? Addition? Element-wise multiplication? Element-wise max? What's the "default"?