0
$\begingroup$

Humans learn facts about the world like "most A are B" by own experience and by being told so (by other people or texts). The systems and mechanisms of storage and usage of such facts (by an "experience system" and a "declarative system") are presumably quite different and may have to do with "episodic memory" and "semantic memory". Nevertheless at least in the human brain the common currency are synaptic weights, and it would be quite interesting to know how these two systems cooperate.

I assume that machine learning is mainly concerned with "learning by own experience" (= training data + annotations), be it supervised or unsupervised learning. I wonder which approaches there are that allow a neural network to "learn by being told". One brute force approach might be to translate a declarative statement like "most A are B" into a set of synthetic training data, but that's definitely not how it works for humans.

Improve this question
$\endgroup$
5
  • $\begingroup$ I would say you are right, in that NNs only learn by being shown an input and an output, ie through 'experience'. "Being told" would map onto pre-defining the weights (ie bypass the training algorithm), but I guess they're too much of a black box for that to be feasible. $\endgroup$
    – Oliver Mason
    Commented Mar 31, 2021 at 8:53
  • $\begingroup$ @OliverMason. Thanks for the comment. I would not say "predefine the weights" or "bypass the training" but to complement/adjust/modulate/correct them (it) - or the other way around: let the training do the fine-tuning. $\endgroup$
    – Hans-Peter Stricker
    Commented Mar 31, 2021 at 9:13
  • $\begingroup$ @OliverMason: To the rest what you say, I fully agree: The weights are too much of a black box to make anything into this direction feasible. (In human brains it works!) $\endgroup$
    – Hans-Peter Stricker
    Commented Mar 31, 2021 at 9:15
  • $\begingroup$ Normally you start with random weights, and the training process adjusts them so that the mapping input to output is achieved; if you pre-define them with non-random values, you could in theory tell the NN to behave in a particular way (if you could choose the correct values). $\endgroup$
    – Oliver Mason
    Commented Mar 31, 2021 at 9:30
  • $\begingroup$ @OliverMason: Does it make a difference to start with some constant weights? $\endgroup$
    – Hans-Peter Stricker
    Commented Mar 31, 2021 at 10:07

1 Answer 1

Reset to default
1
$\begingroup$

One way to look at intelligence is it's the way to compress the universe. That means we have a short mental representation of meaningful concepts.

For example, if I would say "there is a red swan in your building, it's dangerous and can kill you", you already have concepts of "red", "swan", "danger" and this easy allows you to add the bird to your classifier "should I beware it or not". (and you probably have tried to imagine the red swan by now)

There are similar properties in deep networks when deeper layers could respond to more abstract representation concepts. For example, if you have a convolutional network classifying the faces, the first layers would detect simple shapes like lines and circles, then more complex shapes and then parts of the face like eyes, nose and ears (see example here)

Now, let's imagine you have a classification task "is the thing dangerous or not". You have a trained convolutional model and it has "red" and "swan" abstract features on some level. You could freeze all the preliminary weights and make a single backpropagation with a high learning rate only for the subnetwork which is of interest to you.

That could work in theory, but the key challenge here is that most of the neural network representations are not interpretable at all and you would need to solve this problem before (there is a whole research direction with workshops and conferences on the topic with varying success)

Improve this answer
$\endgroup$
2
  • $\begingroup$ The human brain solves this problem easily. How does it do it? $\endgroup$
    – Hans-Peter Stricker
    Commented Mar 31, 2021 at 10:34
  • $\begingroup$ Does human brain solve it easily though? If you told something out of your familiar context, you would probably have hard time to adopt the experience. And if you about the interpretability -- I don't know for sure, though I would speculate it related to multi-task of human and re-usability of concepts. $\endgroup$
    – Kirill Fedyanin
    Commented Mar 31, 2021 at 10:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .