How can we teach a neural net to make arbitrary data associations?

Question

Let's say I have pairs of keys and values of the form $(x_1, y_1), \dots, (x_N, y_N)$. Then I give a neural net a key and a value, $(x_i, y_i)$. For example, $x_i$ could be $4$ and $y_i$ could be $3$, but this does not have to be the case.

Is there a way to teach the neural net to output the $y_i$ variable every time it receives the corresponding $x_i$?

By the way, how do our brains perform this function?

Answer 1

In a nutshell : Memorizing is not Learning

So, first let's just remind the classical use of a neural net, in Supervised Learning :

• You have a set of $(x_{train}, y_{train}) \in X \times Y$ pairs, and you want to extract a general mapping law from $X$ to $Y$
• You use a neural net function $f_{\theta} : x \rightarrow f_{\theta}(x)$, with $\theta$ the weights (parameters) of your net.
• You optimise $f_{\theta}$ by minimizing the prediction error, represented by the loss function.

Can this solve your question ? Well, I don't think so. With this scheme, your neural net will learn an appropriate mapping from the set $X$ to the set $Y$, but this mapping is appropriate according to your loss function , not to your $(x_{train}, y_{train})$ pairs.

Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each $(x_{train}, y_{train})$. So classical Supervised Deep Learning should not memorize $(x_{train}, y_{train})$ pairs.

However, you could memorize using a net with too many parameters : it's Overfitting !

• In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each $(x_{train}, y_{train})$ pair you feed during training.
• However, for an input $x$ that it never saw during training, $f_{\theta}(x)$ would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.

But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be Expert Systems, I don't know them enough to explain them, but you may check that if you want.

What about the brain ?

The matter in answering this question is that we don't really know how does the brain work. I highly recommend this article discussing neural networks and the brain.

Some thoughts to start :

1. The brain has an incredibly huge amount of parameters, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks : so the brain could be also able to overfit, and thus to memorize by this mean.
2. Our brain is not a feed forward network at all, we can't delimitate any layer, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.
3. It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !
4. A more personal thought : the brain is able to both learn and memorize ("The exception that proves the rule" motto shows that I think), while learning and memorizing are antonyms for neural nets...