1
$\begingroup$

I am new to neural networks. I would like to use them as a fitting or forecasting method.

A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the outputs nodes, represents a linear model. Nonlinearity begins to appear in an ANN model when we have hidden nodes, in which a nonlinear function is assigned to the hidden nodes, and using minimization their weights are determined.

How do we choose the non-linear activation function that should be assigned to each hidden neuron?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Choice of activation functions depends a lot on the task that you want to accomplish. $\endgroup$
    – naive
    Commented Oct 13, 2019 at 14:31
  • 1
    $\begingroup$ This will be of help. $\endgroup$
    – naive
    Commented Oct 13, 2019 at 14:34

2 Answers 2

Reset to default
1
$\begingroup$

To know the form of your non-linear function, firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions, weight decay, momentum and dropout etc.

Improve this answer
$\endgroup$
1
$\begingroup$

TL;DR:One does not know ahead of time what hyper-parameters will achieve optimal performance. So what you need is an iterative implementation strategy:

Implementation Strategy

When working with neural networks it is key to make sure that you spend your time wisely. It is possible to spend lots of time on a dead end simply because you made an assumption about your model at the very beginning.

So when selecting activation functions and other hyper parameters don't over think things. That is, get a quick and dirty model up and running and tune from there. From this model you can iterate. For example, you could start with ReLU activations in the hidden layers and as you tune your model you could experiment with other other activations.

That is, the data and the task at hand along with your tuning shape your model. A highly recommended video on this is A. Ng's lecture here and this video from the A. Ng deep learning specialization.

Some content not in the video is how to use learning curves to help define your iterations. These help you decide what you should do next when your model is not achieving desired performance.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Do you know if this things have changed to nowadays? $\endgroup$
    – Verónica Rmz.
    Commented Apr 24, 2021 at 22:51
  • 1
    $\begingroup$ @VerónicaRmz. Deeplearning.ai has just released an updated version of the deep learning specialization. They have one course specifically dedicated to tuning neural networks. I recommend checking it out on Coursera. $\endgroup$
    – respectful
    Commented Apr 26, 2021 at 13:35

Not the answer you're looking for? Browse other questions tagged .