Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar items. Suppose we also have access to embeddings for each item.

A naive approach might be to concat the two item embeddings, add a linear layer or two and finally perform a sigmoid (as this is binary classification) for the output probability.

However, that approach would mean that potentially inputing (x, y) to the model could give a different score from inputing (y, x) into it, since concat is not symmetric.

How can we go about overcoming this? What is the common practice in this situation?

So far I have thought about:

  1. Whenever I input (x, y) I can also input (y, x) and always take the average prediction of both of them. But this feels like a hacky way of forcing the network to be symmetric, it doesn't make it learn the same thing despite of the input order.

  2. Replacing concat with some other symmetric tensor operation. But what operation? Addition? Element-wise multiplication? Element-wise max? What's the "default"?

  • $\begingroup$ Out of interest, why in the example scenario do you have embeddings where "similar" items have dissimilar embeddings? Are you not in control of the encoder? $\endgroup$ Jan 27 at 12:23
  • $\begingroup$ I never said that "similar" items have dissimilar embeddings. The embeddings are fixed and extracted from text with pretrained BERT. My problem is in the ordering that I should give each pair of item embeddings. My intuition says that the order shouldn't matter. $\endgroup$
    – Michael
    Jan 27 at 12:50
  • 1
    $\begingroup$ If you mean to ask "Why do we need supervision for similarity if we can just calculate the similarity on the embeddings?" then I would say that is a good point that had not occured to me. I guess the fact that the embeddings are fixed is a valid reason. The final layer can be "correcting actions" or a way to learn a threshold of "how similar is similar enough". $\endgroup$
    – Michael
    Jan 27 at 13:00
  • 1
    $\begingroup$ Yes that is why I was asking. One of your solutions here - which sidesteps the symmetry issue - is to train such that the embeddings already solve your problem. This is how face recognition usually works. That could be part of an answer, but whether or not is worth mentioning depends on whether you are asking to solve your original problem (classifying pairs as similar), or whether you want to stay focused on the symmetry issue already presented in the question. $\endgroup$ Jan 27 at 14:20
  • $\begingroup$ I don't know whether training embeddings for low distance when similar and high distance when different (triplet loss is a standard example) is used much in text systems. Chances are if you are using an embedding, it will be from some pre-trained general language model, which may be very expensive to train or re-train yourself. $\endgroup$ Jan 27 at 14:22

2 Answers 2


The problem you're describing is related to (if not a subset of) Shift Invariance. Shift invariance refers to any geometric translation of an input, but concatenation of a pair of tenors in 2 different ways $(x, y) \rightarrow (y, x)$ can be seen as translation with step equal to the shape of the tensors.

How to tackle lack of shift invariance? There is still not unanimous consensus on why deep neural network are not shift invariant, even though some papers pointed out that some convolution operations might be a core issue.

  • Zhang proposed an alternative variation of pooling tat should enhance anti-aliased feature maps
  • Chaman & Dokmanic focus instead on analyzing the impact of dawn sampling operations, suggesting a new subsampling operation as well to replace in the conventional down sampling approaches.

Starting from a completely different perspective, other papers analyze the impact of classic euclidean geometry, utilized not only in loss design (l1 and l2 norm for example), but also an underlying assumption of every classic deep learning model (despite non linear activation functions, every hidden layer is still a linear transformation of the form $w*x + b$). So instead of fixing linear operations or enforce good shift invariant feature maps, we change the geometry we're using to ensure that translation and rotations have no impact in the optimization of our objective.

  • This paper is a good start if you feel brave enough to start experimenting with not euclidean approaches.

On a final note, I think that you first suggestion could be worth a shot if you rethink it this way:

  • Present the model each time 2 pairs $(x, y)$ & $(y, x)$ and add a custom loss component to enforce the same prediction (could be literally MSE between the two outputs logits). This loss should at least give you a hint if it's possible for the model to become robust to this specific translation operation (if that loss component decrease over training time the answer is yes).

I answered a similar question here. So the goal here is to train a network which can tell whether the inputs $x$ and $y$ are "similar" or not. You can first build a model $f$ which "compresses" the high-dimensional input into a smaller embedding dimension. In the case of Xception this $f$ would be a mapping from a $299 \times 299 \times 3$ RGB image to a $2048$ "feature vector".

Now the classifier model $c(x, y)$ can be built as $c(x, y) = g(f(x) - f(y))$, where $g$ can be a very simple function without any trainable parameters like $g(\overline{d}) = 1 - e^{-\sum_i d_i^2}$ or something more complex. Clearly here $c(x, y) = c(y, x)$ and $c(x, x) = 0$.

With a custom $f$ and $g$ you can train this model end-to-end, or if your embeddings are fixed (or you use a pre-trained network $f$) it is also possible to just train the $g$.

  • $\begingroup$ So in your version, I would take the difference between the f(x) and f(y) and then pass them through network g(x) instead of doing g(concat(f(x), f(y))? But does the loss changes? Do I still use binary cross entropy for classification or does my task change? $\endgroup$
    – Michael
    Jan 28 at 16:07
  • $\begingroup$ I thought Siamese Networks that employ ranking losses were used to learn embeddings where similar objects are closer than dissimilar objects not to predict the similarity itself. $\endgroup$
    – Michael
    Jan 28 at 16:08
  • $\begingroup$ Also, the difference is not symmetric either so.. problem not solved. $\endgroup$
    – Michael
    Jan 28 at 16:36
  • $\begingroup$ Yes you still use binary cross entropy since your target is still a 0 / 1 prediction. You can take element-wise absolute value (or square them) to guarantee a symmetric model $c$. $\endgroup$
    – NikoNyrh
    Jan 28 at 19:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .