Deep Learning Architecture where outputs from two different inputs are used for error calculation

Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)?

Workflow:

Input1 -----> Model ------> Output1

Input2 -----> Model ------> Output2

Loss = criterion(Output1, Output2)

Backpropagation(Loss)

• There is at least one model training process that I can think of that works like this in practice. I am not sure what the downvote was for, but perhaps you could explain what your original problem is, and what you want to understand about the model. Otherwise the direct answer to your question is "yes" with no details. Dec 20 '21 at 9:07

There are known architectures to implement this idea, namely, seamese networks, and also a training strategy, known as contrastive learning, that relies on the idea of comparing the output of neural networks. I will explain both of them briefly.

The idea of seamese networks is exactly what you mentioned. You have a single model, $$m$$, that receives two inputs, $$x_{1}$$ and $$x_{2}$$. Uppon the application of $$m$$, one has $$h_{i} = m(x_{i})$$.

Now, seamese networks have been employed in at least two situations that I know of. The first one is known as Contrastive Learning, and seeks to learn a classifier from pairs of examples. A pair $$(x_{i}, x_{j})$$ is said to be a positive pair if they belong to the same class, otherwise it is called a negative example. The idea is to encode both examples with $$m$$, then comparing the learned representations through a similarity measure $$sim(h_{i}, h_{j})$$. As an example of similarity measure, take the inner product,

$$sim(h_{i}, h_{j}) = \sum_{k=1}^{p}h_{ik}h_{jk}$$

Using this, the contrastive loss reads as,

$$\ell(h_{i}, h_{j}) = \dfrac{exp(sim(h_{i}, h_{j}) / \tau)}{\sum_{n=1}^{N}exp(sim(h_{i}, h_{n}) / \tau)}$$

where $$\tau > 0$$ is known as the temperature, in allusion to Statistical Mechanics. The principle behind this type of learning strategy is the the model $$m$$ will learn to encoder inputs in such a way that positive pairs (samples from the same class) are mapped close together in the latent space, whereas negative pairs will be far apart. For reference you can consult [He et al., 2020]

The second application that I know of is metric learning. In this case, suppose that you have a metric $$d$$ that is useful, but very costly to calculate over the input space (e.g. $$d(x_{1}, x_{2})$$ takes a lot of time to calculate). This is the case, for instance, with the Wasserstein distance. The idea is to precompute a bunch of values $$d_{ij} = d(x_{i}, x_{j})$$, and to train a seamese network so that $$\lVert h_{i} - h_{j} \rVert \approx d_{ij}$$, that way we may measure $$d_{ij}$$ by computing the Euclidean distance over the latent space. This is roughly the idea behind the work of [Courty et al., 2017]

References

[He et al., 2020] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[Courty et al., 2017] Courty, Nicolas, Rémi Flamary, and Mélanie Ducoffe. "Learning wasserstein embeddings." arXiv preprint arXiv:1710.07457 (2017).