I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are all non linear, and they all can be different. And in the first layer, they are all linear.


On what does this depends?

  1. When it is said "the neural network learns automatically", in colloquial words, what does it mean?

AFAIK, one first train the NN, then at some point NN learns. When does the "automatically" enters then?

Thanks in advance for your help.

  • $\begingroup$ Hello Veronica. Could you please focus on 1 question at a time (so that people can focus on 1 problem at a time)? So, I would suggest that you remove the second question and ask it in a different/separate post. Please, try also to put your specific question in title to give immediately the idea of what your post is about to the readers. $\endgroup$
    – nbro
    Apr 22 '21 at 1:03
  • $\begingroup$ @nbro Hello moderator nbro. Ok, but notice that nxglogic has answered both questions, if I edit my question, that would affect their answer, so?. I've modified the title to make it more specific. $\endgroup$ Apr 22 '21 at 16:00
  • $\begingroup$ Helpful -> ai.stackexchange.com/questions/7088/… and their respective answers. $\endgroup$ Apr 25 '21 at 1:20

Rosenblatt was probably discussing a specific architecture, for which there are many. However, for general purpose feed-forward back-propagation ANNs used for function aproximation and classification analysis, you can use whatever activation functions you want on the input-side, hidden layers, and output-side. Examples are identity, logistic, tanh, exponential, Hermite, Laguerre, RBFN, ReLu, softmax, etc. "Automatically," likely refers to the iterative learning process, which tends to be similar to gradient descent, during which partial derivatives of prediction error w.r.t to coefficients decrease.

  • $\begingroup$ Thank you for your help. $\endgroup$ Apr 22 '21 at 15:58
  • $\begingroup$ On what does depends the choice of the activation functions? $\endgroup$ Apr 22 '21 at 15:59
  • 1
    $\begingroup$ You could take a look at the Chris Bishop book at academia.edu/35719617/… (just page down to see pages). ANNs really do like inputs in the range [-1,1], so feature standardization is a good way to start, either that or normalization to range[0,1] then subtract 0.5 from input values. Given this tanh or logit (lostistic) would be my first choice at the hidden layer, then either identity or logistic at the output-side. Softmax is a good choice for classification on the output-side. $\endgroup$ Apr 22 '21 at 22:34
  • 1
    $\begingroup$ You have to assess error (e.g., MSE, cross-entropy) based on which functions you use. But also, you should use cross-validation as well. $\endgroup$ Apr 22 '21 at 22:34
  • $\begingroup$ Is it common to take a single activation function for the entire layer? $\endgroup$ Apr 23 '21 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.