Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
3
votes
1answer
335 views

Is the cube root function suitable as a n activation function?

I am trying to design a neural network on Python. Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph: Is this ...
3
votes
1answer
106 views

Should the input to the negative log likelihood loss function be probabilities?

I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the ...
2
votes
3answers
137 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
0
votes
1answer
387 views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
6
votes
1answer
75 views

What makes multi-layer neural networks able to perform nonlinear operations?

As I know, a single layer neural network can only do linear operations, but multilayered ones can. Also, I recently learned that finite matrices/tensors, which are used in many neural networks, can ...
10
votes
2answers
168 views

What does it mean for a neuron in a neural network to be activated?

I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what ...
10
votes
1answer
8k views

What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
13
votes
2answers
6k views

How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
4
votes
1answer
145 views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
7
votes
2answers
3k views

Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...
3
votes
2answers
110 views

ANNs with multiple activation outputs

Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of single firing. I'm interested to know if this would have any ...
19
votes
6answers
10k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
2
votes
1answer
732 views

What are the counterparts of non-linearities and dropout in fully convolutional networks?

I am trying to replicate the fully convolutional networks (FCN) concept described here for semantic segmentation. It seems people have successfully trained such models by removing fully connected ...
9
votes
5answers
3k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?

1
2