# What are the prior beliefs in a neural network? (if any)

Let us confine ourselves to the case where we have a $$n$$ dimensional input and a $$+1$$ or $$-1$$ output. It can be shown that:

For every $$n$$, there exists a dense NN of depth 2, such that it contains all functions from $${±1}^n$$ to $${±1}$$. (given sign activation functions and some other very simple assumptions).

Check section 20.3 for the proof.

So, if neural net can approximate any function then it has a $$\mathcal V \mathcal C$$ dimension of infinite (considering the $$n$$ dimesnional set of points as our universe)

Thus it can realise all types of functions or its hypothesis set contains all set of functions and hence cannot have prior knowledge. (prior knowledge in the sense used in the No Free Lunch theorem).

Are my deductions correct? Or did I make mistaken assumptions? Are there actually any prior beliefs in a neural network which I am missing. A detailed explanation would be nice.

• Comments are not for extended discussion; this conversation has been moved to chat. – nbro Apr 13 at 22:39