Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
0
votes
0answers
13 views

How to train with a non-differentiable activation function (such as SVT in deep unrolling low-rank optimization)?

I planned to design a deep unfolding for decomposition into low-rank and sparse in Pytorch environment. I read this paper that might help me to understand how to do it. I always taught that this model ...
2
votes
1answer
65 views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
4
votes
1answer
68 views

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
0
votes
1answer
41 views

Why identity function is generally treated as an activation function?

It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity. Then how can the linear activation function, especially the identity function, be ...
3
votes
1answer
71 views

Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
0
votes
0answers
8 views

What is an intuitive explanation for the weighted sum of inputs plus bias that cause a neuron to be activated when it sees some samples but not others

Im stuck on some of the intuition thats cause specific neurons to fire versus others. Take a feedforward MLP that is able to classify MNIST (and has been optimised). A silly example might be that in ...
0
votes
0answers
45 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
2
votes
2answers
60 views

Does the output layer in a deep neural network need an activation function?

I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows: and here is a PyTorch implementation: <...
1
vote
1answer
53 views

What is meant by "well-behaved gradient" in this context?

Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models So far, the most striking successes in deep learning have involved ...
2
votes
1answer
251 views

What does "linear unit" mean in the names of activation functions?

Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. &...
0
votes
1answer
52 views

Is my understanding on "smooth approximation" correct?

Consider the following details regarding Softplus activation function $$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$ SoftPlus is a smooth approximation to the ReLU function and can be ...
0
votes
0answers
27 views

Why is the activation function "HardShrink" called so?

Neural networks contain activation functions, which are responsible for the non-linearity of their intermediate and final outputs. Hardshrink is the name of an activation function, which is defined ...
1
vote
0answers
31 views

Using a rectified Tanh to train a CNN?

I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. ...
1
vote
0answers
27 views

Is the main difference between the logistic regression and the perceptron the activation function they use?

I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point. I'd like to consider the question in terms of the ...
0
votes
0answers
37 views

tanh activation function output is not between -1 and 1 for continuous action PPO

I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
2
votes
1answer
178 views

Why is tanh a "smoothly" differentiable function?

The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature. The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
0
votes
0answers
15 views

Why doesn't a neuron activation depend on number of input (presynaptic) neurons?

In an artificial neural network, we usually use the same activation function for all neurons, independently of the number of input (presynaptic) neurons. However, usually, the number of input neurons ...
0
votes
0answers
36 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying? (#2)

I recently asked a very similar question here, but the answer only seems to address the first part of the quote, rather than the second part that contains the perceptron criterion example. Therefore, ...
0
votes
1answer
58 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The classical activation ...
1
vote
0answers
16 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
2
votes
3answers
249 views

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
2
votes
1answer
94 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
0
votes
1answer
38 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
0
votes
1answer
53 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
0
votes
0answers
9 views

GAN unexpected behavior depeding on scaling and generator output activation function

I am training a GAN using data that underwent PCA. When I scale the data between -1 and 1, no matter when I use 'tanh' or 'sigmoid' at the last layer of the generator, the network is not stable. ...
1
vote
0answers
30 views

Is there literature on Neural Network with activation functions of bounded domain?

I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
2
votes
1answer
170 views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
0
votes
1answer
60 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
1
vote
2answers
36 views

Trying to understand why nonlinearity is important for neural networks by analogy

Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are ...
0
votes
1answer
83 views

Why can a neural network use more than one activation function?

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
4
votes
2answers
223 views

What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), ...
0
votes
1answer
74 views

Why my Fully Connected Neural Network outputs the same prediction?

I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form ...
3
votes
1answer
163 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
2
votes
1answer
697 views

Why is no activation function needed for the output layer of a neural network for regression?

I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back ...
-1
votes
1answer
31 views

setting up last layer in tensoflow for class type of label [closed]

I am creating a NN in tensorflow keras. the inputs are all float and the output is a class. The output currently encoded as a float, but only has 4 values (0,1,2,3). My model is similar to this: ...
5
votes
0answers
71 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
1
vote
0answers
26 views
1
vote
1answer
135 views

What happens if there is no activation function in some layers of a neural network?

What if I don't apply an activation function on some layers in a neural network. How will it affect the model? Take for instance the following code snippet: ...
2
votes
0answers
37 views

Given a 2-layer GCN, can we choose the dimensions of the 2nd weight matrix, such that this architecture has the same capacity as a 1-layer GCN?

This might be more of a question about nested function classes: For $k$ class node classification in a graph with $n$ nodes, and $d$ feature vector. I want to compare Architecture I: the GCN model of ...
2
votes
1answer
49 views

Which NN would you choose to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$?

Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you ...
1
vote
0answers
25 views

What is the intuition behind equations 10, 11 and 12 of the paper "Noise2Noise: Learning Image Restoration without Clean Data"?

Can anyone help me understand these functions described in the paper Noise2Noise: Learning Image Restoration without Clean Data I have read the portion A.4 in the appendix but need a more detailed and ...
4
votes
0answers
84 views

Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
1
vote
2answers
259 views

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
1
vote
0answers
56 views

What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...
1
vote
1answer
83 views

Can most of the basic machine learning models be easily represented as simple neural network architectures?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following: Different ...
0
votes
1answer
59 views

How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
2
votes
1answer
86 views

Evolved networks fail to solve XOR

My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do ...
2
votes
0answers
80 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
1
vote
0answers
38 views

Dynamically adapting activation function

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...