I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code. I am curious about what is the exact structure within a node of a hidden layer and the areas of customization?
From what I have gathered, each node receives some inputs (either features of your original data or outputs from the previous layer). The first step is, we form a linear combination of the inputs with coefficients in the node. Here's my first few questions:
- For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?
- In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?
- If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?
- In terms of customization, for most out of the box NNs, do we choose to have a bias term or not? Or for the most part, it automatically includes one.
- In terms of customization, this combination of inputs is always linear? There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)
After we have generated a scalar via the linear combination, the second step is we feed that value to our "activation function" which is most often a non-linear function like ReLU. Here obviously the customizable aspect is the choice of "activation" function.
- For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?