Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
2
votes
1answer
29 views

What would be the implications of mistakenly adding bias after the activation function?

I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was ...
2
votes
2answers
64 views

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four ...
2
votes
2answers
95 views

Why is there a sigmoid function in the hidden layer of a neural network? [duplicate]

I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is ...
1
vote
1answer
53 views

When should I use a linear activation instead of ReLU?

I have read this post: How to choose an activation function?. There is enough literature about activation functions, but when should I use a linear activation instead of ReLU? What does the author ...
5
votes
4answers
850 views

Do neurons of a neural network model a linear relationship?

I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics ...
2
votes
2answers
57 views

Can multiple activation functions be replaced with a single activation function?

I'm just started to learn deep learning and I have a question about this neural network: I think $h_1$, $h_j$ and $h_n$ are perceptrons. So, if they are perceptrons, all of them will have an ...
0
votes
1answer
37 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
6
votes
3answers
1k views

Why is the derivative of the activation functions in neural networks important?

I'm new to NN. I am trying to understand some of its foundations. One question that I have is: why the derivative of an activation function is important (not the function itself), and why it's the ...
3
votes
2answers
63 views

Is PReLU superfluous with respect to ReLU?

Why do people use the $PReLU$ activation? $PReLU[x] = ReLU[x] + ReLU[p*x]$ with the parameter $p$ typically being a small negative number. If a fully connected layer is followed by a at least two ...
1
vote
1answer
20 views

Why are activation functions independent layers in CNNs rather than part of convolutional layers?

I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization layers. A common normalization layer is a ReLU layer. But I never encountered an ...
5
votes
1answer
176 views

What's the advantage of log_softmax over softmax?

Previously I have learned that the softmax as the output layer coupled with the log-likelihood cost function (the same as the ...
3
votes
1answer
65 views

What kind of functions can be used as activation functions?

I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an ...
4
votes
1answer
58 views

How do intermediate layers of a trained neural network look like?

Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a ...
2
votes
1answer
131 views

Regarding the output layer's activation function for continuous action space problems

I'm interested in building a (deep) RL agent for solving a continuous problem (which splits something into portions). In all examples I've seen so far, e.g., solving the continuous lunar lander, ...
0
votes
0answers
104 views

Selu and how to implement it?

I just came accros the "Selu" function. It looks promising, especially with the header "self normalizing neural network", since I have some serious with exploding gradient within my network with (...
5
votes
2answers
71 views

What should the range of the output layer be when performing classification?

I am working on a MLP neural networks, using supervised learning (2 classes and multi-class classification problems). For the hidden layers, I am using $\tanh$ (which produces an output in the range $[...
1
vote
0answers
40 views

How do two perceptrons produce different linear decision boundaries when learning?

I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if these two perceptrons converge to two different ...
1
vote
0answers
26 views

Activation Function of the Last Layer

I was wondering whether there is a method (not a table of recommendations) that could tell me what activation function to choose if the outputs of the neural network have some interpretation . For ...
1
vote
1answer
40 views

Is it a great misconception that the softmax is an activation function?

An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation. A softmax however, is a ...
6
votes
1answer
190 views

Why isn't the ElliotSig activation function widely used?

The Softsign (a.k.a. ElliotSig) activation function is really simple: $$ f(x) = \frac{x}{1+|x|} $$ It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally ...
2
votes
1answer
161 views

What is a “logit probability”?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
10
votes
1answer
256 views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
1
vote
1answer
149 views

Would this formula be relevant to the field of A.I?

Background My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another. $$ \sum_i a_i w_i = a'_{k} $$ But to compute this weighted sum the sum ...
1
vote
2answers
247 views

What is the derivative function used in backpropagration?

I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function? These terms are confusing: ...
2
votes
0answers
105 views

Non-monotonic activation function and XOR problem with perceptron

I already post the question on Stackoverflow : https://stackoverflow.com/questions/53785922/is-the-use-of-a-non-monotonic-activation-function-is-a-correct-and-viable-solut?noredirect=1#...
3
votes
3answers
254 views

Which neural network should I use to approximate a specific function?

We have convolutional neural networks and recurrent neural networks for analysing respectively images and sequential data. How do I determine which neural network architecture is more appropriate to ...
7
votes
4answers
328 views

Is it suitable to find inverse of last layer's activation function and apply it on the target output?

I have a neural network with the following structure: I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the ...
3
votes
1answer
237 views

Is the cube root function suitable as a n activation function?

I am trying to design a neural network on Python. Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph: Is this ...
3
votes
1answer
69 views

Should the input to the negative log likelihood loss function be probabilities?

I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the ...
2
votes
3answers
131 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
0
votes
1answer
244 views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
10
votes
2answers
135 views

What does it mean for a neuron in a neural network to be activated?

I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what ...
9
votes
1answer
5k views

What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
11
votes
2answers
4k views

How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
3
votes
1answer
132 views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
3
votes
2answers
103 views

ANNs with multiple activation outputs

Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of single firing. I'm interested to know if this would have any ...
15
votes
6answers
7k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
2
votes
1answer
690 views

What are the counterparts of non-linearities and dropout in fully convolutional networks?

I am trying to replicate the fully convolutional networks (FCN) concept described here for semantic segmentation. It seems people have successfully trained such models by removing fully connected ...
8
votes
5answers
2k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?