0
$\begingroup$

I am new to neural networks. I would like to use them as a fitting or forecasting method.

A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the outputs nodes, represents a linear model. Nonlinearity begins to appear in an ANN model when we have hidden nodes, in which a nonlinear function is assigned to the hidden nodes, and using minimization their weights are determined.

How do we choose the non-linear activation function that should be assigned to each hidden neuron?

$\endgroup$
  • $\begingroup$ Choice of activation functions depends a lot on the task that you want to accomplish. $\endgroup$ – naive Oct 13 '19 at 14:31
  • $\begingroup$ This will be of help. $\endgroup$ – naive Oct 13 '19 at 14:34
0
$\begingroup$

To know the form of your non-linear function, firstly you should define the type of problem you are dealing with such as an image classification task. Secondly, pick the activation functions based on your task such as sigmoid, Tanh, ReLu, LeadyRelu, Softmax etc. Overall, your ANN performance mainly depends on the number of hidden layers (hidden units), selection of activation functions, weight decay, momentum and dropout etc.

$\endgroup$
0
$\begingroup$

TL;DR:One does not know ahead of time what hyper-parameters will achieve optimal performance. So what you need is an iterative implementation strategy:

Implementation Strategy

When working with neural networks it is key to make sure that you spend your time wisely. It is possible to spend lots of time on a dead end simply because you made an assumption about your model at the very beginning.

So when selecting activation functions and other hyper parameters don't over think things. That is, get a quick and dirty model up and running and tune from there. From this model you can iterate. For example, you could start with ReLU activations in the hidden layers and as you tune your model you could experiment with other other activations.

That is, the data and the task at hand along with your tuning shape your model. A highly recommended video on this is A. Ng's lecture here and this video from the A. Ng deep learning specialization.

Some content not in the video is how to use learning curves to help define your iterations. These help you decide what you should do next when your model is not achieving desired performance.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.