Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
1
vote
0answers
41 views
+50

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
2
votes
1answer
59 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
1
vote
1answer
83 views

Why don't ELUs multiply the linear portion by $\alpha$?

An exponential linear unit (as proposed by Clevert et al.) uses the function: \begin{align} \text{ELU}_\alpha(x) = \begin{cases} \alpha(e^x - 1), &\text{if } x < 0\\ x, \text{if} &\text{if ...
0
votes
1answer
40 views

Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
0
votes
1answer
34 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
2
votes
1answer
170 views

How to choose the activation function in neuroevolution?

I am developing a NEAT flappy bird game, and it doesn't work, the system stays stupid for 300 generations. I chose tanh() for activation, just because it's included in JS. I can't find a good ...
0
votes
0answers
8 views

GAN unexpected behavior depeding on scaling and generator output activation function

I am training a GAN using data that underwent PCA. When I scale the data between -1 and 1, no matter when I use 'tanh' or 'sigmoid' at the last layer of the generator, the network is not stable. ...
0
votes
1answer
60 views

Why can a neural network use more than one activation function?

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
1
vote
0answers
27 views

Is there literature on Neural Network with activation functions of bounded domain?

I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
5
votes
4answers
2k views

Do neurons of a neural network model a linear relationship?

I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics ...
2
votes
1answer
89 views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
20
votes
3answers
9k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
0
votes
0answers
40 views

What are the most used and effective activation functions for sentiment classification with an recurrent neural network?

I am making an RNN for sentiment classification. What activation functions would you use in order to achieve this goal (excluding the one present in the output layer)?
0
votes
1answer
49 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
4
votes
2answers
69 views

What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), ...
1
vote
2answers
32 views

Trying to understand why nonlinearity is important for neural networks by analogy

Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are ...
0
votes
1answer
51 views

Why my Fully Connected Neural Network outputs the same prediction?

I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form ...
30
votes
5answers
17k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
0
votes
2answers
103 views

Why is non-linearity desirable in a neural network?

Why is non-linearity desirable in a neural network? I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear ...
2
votes
1answer
129 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
2
votes
1answer
84 views

Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?

The associative property of multidimensional discrete convolution says that: $$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$ where $h_1$ and $h_2$ are the filters and $x$ is ...
2
votes
1answer
195 views

Why is no activation function needed for the output layer of a neural network for regression?

I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back ...
1
vote
1answer
281 views

What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...
0
votes
1answer
25 views

setting up last layer in tensoflow for class type of label [closed]

I am creating a NN in tensorflow keras. the inputs are all float and the output is a class. The output currently encoded as a float, but only has 4 values (0,1,2,3). My model is similar to this: ...
3
votes
1answer
254 views

What is the simplest classification problem which cannot be solved by a perceptron?

What is the simplest classification problem which cannot be solved by a perceptron (that is a single-layered feed-forward neural network, with no hidden layers and step activation function), but it ...
0
votes
0answers
26 views

Are all activation functions in a layer same? [duplicate]

I understand that for you can have multiple activation functions in different layers. CNN's usually have Relu followed by softmax for the classification. But what stops us in having multiple ...
5
votes
2answers
737 views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
4
votes
0answers
57 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
2
votes
2answers
218 views

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
1
vote
0answers
20 views

What does Dice Loss should receive in case of binary segmentation

I implemented Dice loss class in pytorch: ...
2
votes
0answers
156 views

If we use a perceptron with a non-monotonic activation function, can it solve the XOR problem?

I found several papers about how to build a perceptron able to solve the XOR problem. The papers describe a solution where the heaviside step function is replaced by a non-monotonic activation ...
5
votes
5answers
644 views

Why can't the XOR linear inseparability problem be solved with one perceptron like this?

Consider a perceptron where $w_0=1$ and $w_1=1$: Now, suppose that we use the following activation function \begin{align} f(x)= \begin{cases} 1, \text{ if }x =1\\ 0, \text{ otherwise} \end{cases} \...
8
votes
3answers
1k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...
3
votes
1answer
242 views

Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
6
votes
1answer
3k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
1
vote
1answer
59 views

What happens if there is no activation function in some layers of a neural network?

What if I don't apply an activation function on some layers in a neural network. How will it affect the model? Take for instance the following code snippet: ...
2
votes
0answers
27 views

Graph Neural Networks: Quesitons about different GCN Architectures

This might be moreof a question about nested function classes: For k class node classification in a graph with n nodes, and d feature vector. I want to compare Architecture I: the GCN model of Kipf/ ...
2
votes
1answer
44 views

Which NN would you choose to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$?

Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you ...
2
votes
0answers
62 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
1
vote
1answer
1k views

Is a linear activation function (in the output layer) equivalent to an identity function?

I have a simple question about the choice of activation function for the output layer in feed-forward neural networks. I have seen several codes where the choice of the activation function for the ...
1
vote
0answers
18 views

What is the intuition behind equations 10, 11 and 12 of the paper “Noise2Noise: Learning Image Restoration without Clean Data”?

Can anyone help me understand these functions described in the paper Noise2Noise: Learning Image Restoration without Clean Data I have read the portion A.4 in the appendix but need a more detailed and ...
4
votes
0answers
74 views

Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
1
vote
2answers
124 views

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
1
vote
0answers
43 views

What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...
3
votes
2answers
2k views

Why is no activation function used at the final layer of super-resolution models?

I'm trying to implement some image super-resolution models on medical images. After reading a set of papers, I found that none of the existing models use any activation function for the last layer. ...
1
vote
1answer
77 views

Can entire neural networks be composed of only activation functions?

Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples. Adversarial iRL, build upon GAN-...
4
votes
1answer
226 views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
6
votes
1answer
387 views

What is the mathematical definition of an activation function? [duplicate]

What is the mathematical definition of an activation function to be used in a neural network? So far I did not find a precise one, summarizing which criterions (e.g. monotonicity, differentiability, ...
1
vote
1answer
63 views

Can most of the basic machine learning models be easily represented as simple neural network architectures?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following: Different ...
0
votes
1answer
35 views

How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?