People who code: we want your input. Take the Survey

# Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

119 questions
Filter by
Sorted by
Tagged with
39 views

### How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
34 views

### What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
38 views

### Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?

I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
8 views

### GAN unexpected behavior depeding on scaling and generator output activation function

I am training a GAN using data that underwent PCA. When I scale the data between -1 and 1, no matter when I use 'tanh' or 'sigmoid' at the last layer of the generator, the network is not stable. ...
27 views

### Is there literature on Neural Network with activation functions of bounded domain?

I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
89 views

### Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
49 views

### About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
32 views

### Trying to understand why nonlinearity is important for neural networks by analogy

Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are ...
60 views

### Why can a neural network use more than one activation function?

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
69 views

### What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), ...
51 views

### Why my Fully Connected Neural Network outputs the same prediction?

I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form ...
128 views

### Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
183 views

### Why is no activation function needed for the output layer of a neural network for regression?

I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back ...
25 views

### setting up last layer in tensoflow for class type of label [closed]

I am creating a NN in tensorflow keras. the inputs are all float and the output is a class. The output currently encoded as a float, but only has 4 values (0,1,2,3). My model is similar to this: ...
26 views

### Are all activation functions in a layer same? [duplicate]

I understand that for you can have multiple activation functions in different layers. CNN's usually have Relu followed by softmax for the classification. But what stops us in having multiple ...
57 views

### Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
20 views

### What does Dice Loss should receive in case of binary segmentation

I implemented Dice loss class in pytorch: ...
59 views

### What happens if there is no activation function in some layers of a neural network?

What if I don't apply an activation function on some layers in a neural network. How will it affect the model? Take for instance the following code snippet: ...
27 views

### Graph Neural Networks: Quesitons about different GCN Architectures

This might be moreof a question about nested function classes: For k class node classification in a graph with n nodes, and d feature vector. I want to compare Architecture I: the GCN model of Kipf/ ...
44 views

### Which NN would you choose to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$?

Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you ...
18 views

### What is the intuition behind equations 10, 11 and 12 of the paper “Noise2Noise: Learning Image Restoration without Clean Data”?

Can anyone help me understand these functions described in the paper Noise2Noise: Learning Image Restoration without Clean Data I have read the portion A.4 in the appendix but need a more detailed and ...
74 views

### Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
123 views

### How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
43 views

### What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...
62 views

### Can most of the basic machine learning models be easily represented as simple neural network architectures?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following: Different ...
35 views

### How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
18 views

### How to afine the extremity values in regression prediction with Keras?

I made a stack of bidirectional LSTM layers following by Dense layers (with swish activation functions) in order to predict a continuous value between 0 and 2. I compiled the model with ...
65 views

### Evolved networks fail to solve XOR

My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do ...
62 views

### How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
35 views

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
137 views

### Smallest possible network to approximate the $sin$ function

The main goal is: Find the smallest possible neural network to approximate the $sin$ function. Moreover, I want to find a qualitative reason why this network is the smallest possible network. I have ...
95 views

### Which hyperparameters in neural network are accesible to users adjustment

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like: amount of hidden layers amount of ...
19 views

### Why is sine activation function not used frequently since we know from fourier transforms that sine functions can combine to fit any function?

Pretty much the title. I'm no expert but from what I know, if you add up enough sine functions with proper amplitudes and frequencies you can get any function you want as a result. With that knowledge,...
65 views

### Do we need non-linear activation function in neural networks whose task isn't classification?

While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable. So I wonder, if we have ...
77 views

### Can entire neural networks be composed of only activation functions?

Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples. Adversarial iRL, build upon GAN-...
277 views

### What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...
40 views

### What are the most used and effective activation functions for sentiment classification with an recurrent neural network?

I am making an RNN for sentiment classification. What activation functions would you use in order to achieve this goal (excluding the one present in the output layer)?
43 views

### Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
64 views

### What is the exact structure within the nodes of a hidden layer? [closed]

I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code. I am curious about what is the exact structure within a node of a hidden layer and ...
65 views

### In what situations ELUs should be used instead of RELUs?

I always use RELUs actication functions when I need to and I understand limitations of ELUs. So in what situation do I need to consider ELUs over RELUs?
89 views

### What activation functions are currently popular?

I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? e.g. ReLU, ELUs, etc. I'd ...
238 views

### Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
87 views

### Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
84 views

### Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?

The associative property of multidimensional discrete convolution says that: $$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$ where $h_1$ and $h_2$ are the filters and $x$ is ...
103 views

### Why is non-linearity desirable in a neural network?

Why is non-linearity desirable in a neural network? I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear ...
101 views

### Why don't we use trigonometric functions for the output neurons?

Why don't we use a trigonometric function, such as $\tan(x)$, where $x$ is an element of the interval $[0,pi/2)$, instead of the sigmoid function for the output neurons (in the case of classification)?...
115 views

### Why not replacing the simple linear functions that neurons compute with more complex functions?

In a neural network, a neuron typically computes a linear function $f(x) = w*x$, where $w$ is the weight and $x$ is the input. Why not replacing the linear function with more complex functions, such ...
44 views

### If the output of a model is a ridge function, what should the activation functions at all the nodes be?

I have the following assignment. I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped. I ...
67 views

### What work has been done with Poisson-style regression via neural networks with exponential activation function?

The first neural net I wrote was a classifier. After that, I learned that neural nets can be used for regression tasks, even quantile regression. It has become clear to me that the usual games with ...
109 views

### Which activation functions should I use for polynomial regression?

I am a beginner in machine learning and neural networks. I have only used neural networks for classification problems. My aim is to modify it so that it can work for polynomial regression as well. In ...