The main characteristic of an activation function is to bring a non linearity property into the NN. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function which range correspond to what you want.
For regression, you usually re-scale your output data to [-1,1] or [0,1] and you use a tanh or sigmoid function in the last layer
For classification, you want to obtain probabilities : use a softmax function in the last layer.
For the hidden layers some functions are better than others :
- The gradient should be fast to compute (from the perspective of your computer).
- If you use too much layer you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.
In practice the Relu function defined by f(x)=max(0, x) works very well and is very simple.