0
$\begingroup$

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are all non linear, and they all can be different. And in the first layer, they are all linear.

Why?

On what does this depends?

  1. When it is said "the neural network learns automatically", in colloquial words, what does it mean?

AFAIK, one first train the NN, then at some point NN learns. When does the "automatically" enters then?

Thanks in advance for your help.

Improve this question
$\endgroup$
3
  • $\begingroup$ Hello Veronica. Could you please focus on 1 question at a time (so that people can focus on 1 problem at a time)? So, I would suggest that you remove the second question and ask it in a different/separate post. Please, try also to put your specific question in title to give immediately the idea of what your post is about to the readers. $\endgroup$
    – nbro
    Commented Apr 22, 2021 at 1:03
  • $\begingroup$ @nbro Hello moderator nbro. Ok, but notice that nxglogic has answered both questions, if I edit my question, that would affect their answer, so?. I've modified the title to make it more specific. $\endgroup$
    – Verónica Rmz.
    Commented Apr 22, 2021 at 16:00
  • $\begingroup$ Helpful -> ai.stackexchange.com/questions/7088/… and their respective answers. $\endgroup$
    – Verónica Rmz.
    Commented Apr 25, 2021 at 1:20

1 Answer 1

Reset to default
1
$\begingroup$

Rosenblatt was probably discussing a specific architecture, for which there are many. However, for general purpose feed-forward back-propagation ANNs used for function aproximation and classification analysis, you can use whatever activation functions you want on the input-side, hidden layers, and output-side. Examples are identity, logistic, tanh, exponential, Hermite, Laguerre, RBFN, ReLu, softmax, etc. "Automatically," likely refers to the iterative learning process, which tends to be similar to gradient descent, during which partial derivatives of prediction error w.r.t to coefficients decrease.

Improve this answer
$\endgroup$
5
  • $\begingroup$ On what does depends the choice of the activation functions? $\endgroup$
    – Verónica Rmz.
    Commented Apr 22, 2021 at 15:59
  • 1
    $\begingroup$ You could take a look at the Chris Bishop book at academia.edu/35719617/… (just page down to see pages). ANNs really do like inputs in the range [-1,1], so feature standardization is a good way to start, either that or normalization to range[0,1] then subtract 0.5 from input values. Given this tanh or logit (lostistic) would be my first choice at the hidden layer, then either identity or logistic at the output-side. Softmax is a good choice for classification on the output-side. $\endgroup$
    – user46301
    Commented Apr 22, 2021 at 22:34
  • 1
    $\begingroup$ You have to assess error (e.g., MSE, cross-entropy) based on which functions you use. But also, you should use cross-validation as well. $\endgroup$
    – user46301
    Commented Apr 22, 2021 at 22:34
  • $\begingroup$ Is it common to take a single activation function for the entire layer? $\endgroup$
    – Verónica Rmz.
    Commented Apr 23, 2021 at 18:32
  • $\begingroup$ Not sure about mixing different activation functions at the same layer for a straighforward ANN; but you would have to have different partial derivatives which could be a nightmare programming. (I don't use open source software). $\endgroup$
    – user46301
    Commented Apr 23, 2021 at 22:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .