Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

1
vote
1answer
33 views

Is it a great misconception that the softmax is an activation function?

An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation. A softmax however, is a ...
1
vote
0answers
27 views

Activation function: why ElliotSig / Softsign is not widely used?

The Softsign (a.k.a. ElliotSig) activation function is really simple: x f(x) = ----------- 1 + |x| It is bounded [-1,1], has a first ...
1
vote
1answer
36 views

What is a “logit probability”?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
8
votes
1answer
82 views

What happens when I mix activation functions?

Hello I am new to neural network and I have a question about activation functions. People usually use their activation function such as relu, sigmoid, tanh etc. But what happens when I mix activation ...
1
vote
1answer
133 views

Would this formula be relevant to the field of A.I?

Background My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another. $$ \sum_i a_i w_i = a'_{k} $$ But to compute this weighted sum the sum ...
1
vote
2answers
222 views

What is the derivative function used in backpropagration?

I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function? These terms are confusing: ...
2
votes
0answers
71 views

Non-monotonic activation function and XOR problem with perceptron

I already post the question on Stackoverflow : https://stackoverflow.com/questions/53785922/is-the-use-of-a-non-monotonic-activation-function-is-a-correct-and-viable-solut?noredirect=1#...
0
votes
2answers
76 views

Neural network architecture for function approximation

We have convolutional neural networks and recurrent neural networks for analysing images and sequential data, respectively. What is the main architecture used for function approximation? (e.g. a ...
6
votes
4answers
160 views

Is it suitable to find inverse of last layer's activation function and apply it on the target output?

I have a neural network with the following structure: I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the ...
3
votes
1answer
66 views

Is the cube root function suitable as a n activation function?

I am trying to design a neural network on Python. Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph: Is this ...
3
votes
1answer
35 views

Is there any problem in training a supervised model with loss function as negative log likelihood loss without using softmax or log softmax?

I am trying to train a supervised model where the output from the model is output of a linear function(WX + b). Kindly note that I'm not using any softmax or log_softmax on the result of the linear. I ...
2
votes
3answers
124 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
0
votes
1answer
89 views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
8
votes
2answers
99 views

What does it mean for a neuron in a neural network to be activated?

I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what ...
6
votes
1answer
1k views

What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
3
votes
2answers
84 views

ANNs with multiple activation outputs

Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of single firing. I'm interested to know if this would have any ...
7
votes
5answers
1k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?