# Can prior knowledge be encoded in deep neural networks?

I was reading Gary Marcus's a Critical Appraisal of Deep Learning. And one of his criticisms is that neural networks don't incorporate prior knowledge in tackling a problem. My question is, has there been any attempts at encoding prior knowledge in deep neural networks?

• Learning in a NN is iterative, thus, based in prior knowledge. In particular, initial value of NN parameters is obviously a prior knowledge. – pasaba por aqui Feb 15 '18 at 16:20
• @pasabaporaqui thanks a lot for the feedback. What my question is driving at is, in most real life problems we have additional information about the domain. Is there a way to inject this prior domain knowledge to improve the success of the model? – Seth Simba Feb 15 '18 at 17:21
• Strictly talking, a neural net doesn't learns, the process of learning is done outside the net (i.e. back propagation algorithm). This is one of the reason is not correct say that a NN simulates brain (in brain, learning is done inside the net). If it doesn't learns, it doesn't use prior knowledge. Prior knowledge can be used to optimize the learning algorithm, including the initial net parameters, not the net behavior itself. If one rule is included in the net (as output and using it in the error function), it is not prior knowledge but a problem constrain that the net will try to fulfill . – pasaba por aqui Feb 15 '18 at 19:50
• Lookup LSTM in relation to neural networks-- it's a method to take advantage of prior knowledge – antlersoft Feb 15 '18 at 19:55
• If learning is the accumulation of knowledge by a determinable process in the present then the past is a memory of the application of this process but the future is a judgement of memory and its difference in comparison, and not a determinable product of knowledge unless we observe an indifference in the past and the future. – Bobs Jul 8 '18 at 15:27

Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.

Yes, we can do it in a deep learner. For example, suppose we have an input vector likes (a, b) and from prior knowledge, we know a^2 + b^2 is important too. Hence, we can add this value to the vectors likes (a, b, a^2 + b^2). As another example, suppose date time is important in your data, but not encoded in the input vector. We can add this to the input vector as a third dimension.

In sum, depends on the structure of the prior knowledge, we can encode it into the input vector.

• I think the questioner was looking for something like LSTM rather than ways the network designer could incorporate their own prior knowledge. – antlersoft Feb 15 '18 at 19:56