1
$\begingroup$

I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code. I am curious about what is the exact structure within a node of a hidden layer and the areas of customization?

From what I have gathered, each node receives some inputs (either features of your original data or outputs from the previous layer). The first step is, we form a linear combination of the inputs with coefficients in the node. Here's my first few questions:

  • For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?
  • In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?
  • If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?
  • In terms of customization, for most out of the box NNs, do we choose to have a bias term or not? Or for the most part, it automatically includes one.
  • In terms of customization, this combination of inputs is always linear? There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)

After we have generated a scalar via the linear combination, the second step is we feed that value to our "activation function" which is most often a non-linear function like ReLU. Here obviously the customizable aspect is the choice of "activation" function.

  • For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?
$\endgroup$
  • $\begingroup$ I am sorry, but I need to close this post because you are asking too many questions. Please, ask only one question per post, even if they are somehow related. If you have multiple questions, ask them in their separate post. Have a look at ai.stackexchange.com/help/on-topic. $\endgroup$ – nbro Sep 9 at 10:19
0
$\begingroup$

Let's go part-by-part.

For a NODE within a layer, if we have p inputs, we would multiply a 1xp coefficient matrix with a px1 input vector, correct?

In practice, for that LAYER, we would multiply a mxp coefficient matrix by a px1 input vector, where m is the number of nodes within that layer?

These two are correct. No issues here. The only differences may be in implementation, the logic here is sound.

If we have a bias term, then it would be a mx(p+1) coefficient matrix and a (p+1)x1 input vector?

This is correct. Intuitively, it is analogous to "offest" term in the equation of a line, y = mx + c. c here plays the same role as the bias.

In terms of customization, for most out of the box NNs, do we choose to have a bias term or not? Or for the most part, it automatically includes one.

Most out-of-the box NN include a bias term by default, since they do provide flexibility to the network, no matter how small. Practically, their utility varies with architecture and application. For example, Feed Forward networks have a good boost in performance with biases when used for regression. CNNs used for classification usually aren't affected much if biases are removed.

In terms of customization, this combination of inputs is always linear? There are no other ways to combine the inputs (such as including polynomials, interaction terms, etc...)

Non-linear combinations are definately possible. But using such combinations basically kills the purpose of ANNs - to have simple stacked operations approximate a complex function. Non-linear combinations may help in rare cases, but more often than not, will probably lead to underfit or overfit models. By convention, any non-linearity is introduced into the model on top of the linear combination, as the activation function.

For each node within the same hidden layer, are the activation functions usually the same, or differ among each node?

Conventionally, all nodes in the same layer have the same activation. This makes vectorization easy, and ultimately doesn't really affect preformance. Instead of varying activations of individual neurons, complex architectures use parallel layers having different activations but with the same inputs. Usually, all stacked layers in models share the same activations, except maybe the last layer and in some cases, the penultimate layer.

| improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.