Let's say I have pairs of keys and values of the form $(x_1, y_1), \dots, (x_N, y_N)$. Then I give a neural net a key and a value, $(x_i, y_i)$. For example, $x_i$ could be $4$ and $y_i$ could be $3$, but this does not have to be the case.

Is there a way to teach the neural net to output the $y_i$ variable every time it receives the corresponding $x_i$?

By the way, how do our brains perform this function?

  • $\begingroup$ Why do you need a neural network to perform this extremely simple task of outputting 3 whenever it receives 4? Can you provide more details about your problem? For example, do you have multiple different parts of numbers or just one pair or what? Also, ask only one question per post, i.e. what is your main question? Is that about mapping 4 to 3 or about brains? Moreover, have you ever heard of supervised learning? $\endgroup$ – nbro Aug 20 '20 at 9:58
  • $\begingroup$ So, I know there are way more simple ways to do this, but I am just curious about how to replicate how our brains memorize things in a neural net. And it wouldn’t be just mapping 4 to 3. It would be any arbitrary association of numbers, like how our brains can memorize anything $\endgroup$ – iamPres Aug 20 '20 at 11:54
  • $\begingroup$ The answer below is correct but doesn't answer the question I feel. Look into hopfield nets and RBMs for the purpose you mention. $\endgroup$ – DuttaA Aug 20 '20 at 13:52
  • $\begingroup$ @DuttaA as far as I know, RBMs do unsupervised learning (visible neurons are for the X data only). If I'm wrong, could you developp your point in an answer ? I'd be pleased to read you. $\endgroup$ – 16Aghnar Aug 23 '20 at 13:11
  • 1
    $\begingroup$ @16Aghnar the main idea behind associative learning is that some neurons fire together. page.mi.fu-berlin.de/rojas/neural/index.html the chapter on associative learning deals with this. I can't point out the exact algo, but it is probably possible to force 2 neurons to fire simultaneously (which probably is a supervised learning paradigm after all). So RBMs was not a good example but modified hopfield nets probably can be modified to do this. The correct term would be hetero associative nets/memory. $\endgroup$ – DuttaA Aug 23 '20 at 13:56

In a nutshell : Memorizing is not Learning

So, first let's just remind the classical use of a neural net, in Supervised Learning :

  • You have a set of $(x_{train}, y_{train}) \in X \times Y$ pairs, and you want to extract a general mapping law from $X$ to $Y$
  • You use a neural net function $f_{\theta} : x \rightarrow f_{\theta}(x)$, with $\theta$ the weights (parameters) of your net.
  • You optimise $f_{\theta}$ by minimizing the prediction error, represented by the loss function.

Can this solve your question ? Well, I don't think so. With this scheme, your neural net will learn an appropriate mapping from the set $X$ to the set $Y$, but this mapping is appropriate according to your loss function , not to your $(x_{train}, y_{train})$ pairs.

Imagine that a small part of the data is wrongly labelled. A properly trained net learns to extract relevant features and thus will predict correctly the label, not like you did. So the net doesn't memorize your pairs, it infers a general law from the data, and this law may not respect each $(x_{train}, y_{train})$. So classical Supervised Deep Learning should not memorize $(x_{train}, y_{train})$ pairs.

However, you could memorize using a net with too many parameters : it's Overfitting !

  • In this case, you set up the net with too many parameters. That gives too much degrees of freedom to your net, and the net will use these DoFs to exactly fit rightly each $(x_{train}, y_{train})$ pair you feed during training.
  • However, for an input $x$ that it never saw during training, $f_{\theta}(x)$ would have no meaning. That's why we say an overfitted net did not learn, and overfitting is feared by many DL practitioner.

But as long as you want only to memorize, and not to learn, a overfitted net may be the a solution. An other solution for memorization may be Expert Systems, I don't know them enough to explain them, but you may check that if you want.

What about the brain ?

The matter in answering this question is that we don't really know how does the brain work. I highly recommend this article discussing neural networks and the brain.

Some thoughts to start :

  1. The brain has an incredibly huge amount of parameters, and has a great plasticity. In that sense, we could draw a parallel with overfitted neural networks : so the brain could be also able to overfit, and thus to memorize by this mean.
  2. Our brain is not a feed forward network at all, we can't delimitate any layer, just some rough zones where we know that some specific information is processed. This makes any parallel between neural nets and the brain difficult.
  3. It's still unclear how our brain updates itself. There's no backpropagation for instance. Our overfitted networks also stem from the update processes (for instance, adding regularization to the loss helps avoiding underfitting), but we have no idea of how this works in the brain, so that's another hurdle to drawing parallels !
  4. A more personal thought : the brain is able to both learn and memorize ("The exception that proves the rule" motto shows that I think), while learning and memorizing are antonyms for neural nets...
  • 1
    $\begingroup$ How can you learn without memorizing? $\endgroup$ – nbro Aug 20 '20 at 9:53
  • $\begingroup$ ahah good question :) your comment asks for a proper definition of memorizing (and maybe I should have done it in my answer). If you consider memorizing as just 'storing the raw input data', then it has little to do with learning, as learning is a statistical optimization problem (for neural nets). When you learn, what you memorize are patterns, trends that you extracted from the raw data. $\endgroup$ – 16Aghnar Aug 20 '20 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.