Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
1
vote
1answer
16 views

Do we need non-linear activation function in neural networks whose task isn't classification?

While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable. So I wonder, if we have ...
0
votes
1answer
52 views

Can entire neural networks be composed of only activation functions?

Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples. Adversarial iRL, build upon GAN-...
0
votes
0answers
22 views

What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...
0
votes
0answers
31 views

What are the most used and effective activation functions for sentiment classification with an recurrent neural network?

I am making an RNN for sentiment classification. What activation functions would you use in order to achieve this goal (excluding the one present in the output layer)?
2
votes
0answers
37 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
1
vote
1answer
58 views

What is the exact structure within the nodes of a hidden layer? [closed]

I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code. I am curious about what is the exact structure within a node of a hidden layer and ...
1
vote
2answers
64 views

In what situations ELUs should be used instead of RELUs?

I always use RELUs actication functions when I need to and I understand limitations of ELUs. So in what situation do I need to consider ELUs over RELUs?
3
votes
1answer
80 views

What activation functions are currently popular?

I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? e.g. ReLU, ELUs, etc. I'd ...
3
votes
1answer
145 views

Why does every neuron in a multi-layer perceptron typically have the same activation function?

Why does every neuron in a multi-layer perceptron typically have the same activation function? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?
3
votes
1answer
58 views

Can residual neural networks use other activation functions different from ReLU?

In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic ...
2
votes
1answer
45 views

Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?

The associative property of multidimensional discrete convolution says that: $$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$ where $h_1$ and $h_2$ are the filters and $x$ is ...
0
votes
1answer
60 views

Why is non-linearity desirable in a neural network?

Why is non-linearity desirable in a neural network? I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear ...
1
vote
2answers
62 views

Why don't we use trigonometric functions for the output neurons?

Why don't we use a trigonometric function, such as $\tan(x)$, where $x$ is an element of the interval $[0,pi/2)$, instead of the sigmoid function for the output neurons (in the case of classification)?...
2
votes
1answer
104 views

Why not replacing the simple linear functions that neurons compute with more complex functions?

In a neural network, a neuron typically computes a linear function $f(x) = w*x$, where $w$ is the weight and $x$ is the input. Why not replacing the linear function with more complex functions, such ...
1
vote
0answers
33 views

If the output of a model is a ridge function, what should the activation functions at all the nodes be?

I have the following assignment. I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped. I ...
2
votes
0answers
40 views

What work has been done with Poisson-style regression via neural networks with exponential activation function?

The first neural net I wrote was a classifier. After that, I learned that neural nets can be used for regression tasks, even quantile regression. It has become clear to me that the usual games with ...
2
votes
0answers
49 views

Which activation functions should I use for polynomial regression?

I am a beginner in machine learning and neural networks. I have only used neural networks for classification problems. My aim is to modify it so that it can work for polynomial regression as well. In ...
1
vote
1answer
39 views

Are activation functions applied to feature maps?

If I have a convolutional neural network, and I convolve my input tensor with a kernel, the output is a feature map. Is an activation function then applied to this feature map? If its an image that ...
1
vote
0answers
36 views

Can SqueezeNet be used for regression?

I want a model that outputs the pixel coordinates of the tip of my forefinger, and whether it's touching something or not. Those would be 3 output neurons: 2 for the X-Y coordinates and 1, with a ...
1
vote
2answers
73 views

How are non-linear surfaces formed in the training of a neural network?

Desperate trying to understand something for couple of weeks. All those questions are actually one big question.Please help me. Time-codes and screens in my question refer to this great(IMHO) 3d ...
2
votes
0answers
30 views

Single label classification into hierarchical categories using a neural network

I am working on a classification problem into progressive classes. In other words, there is some hierarchy of categories in such a way, that A < B < C, e.g. low, medium, high, very high. What ...
3
votes
0answers
43 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
1
vote
0answers
46 views

Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
2
votes
2answers
98 views

Is ReLU a non-linear activation function?

According to this blog post The purpose of an activation function is to add some kind of non-linear property to the function The sigmoid is typically used as an activation function of a unit of a ...
2
votes
3answers
137 views

Using sigmoid in LSTM network for multi-step forecasting

I'm trying to develop a multistep forecasting model using LSTM Network. The model takes three times steps as input and predicting two time_steps. both input and output columns are normalised using ...
1
vote
0answers
42 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
0answers
32 views

What are the pros and cons of the common activation functions?

I have heard that sigmoid activation functions should not be used on neural networks with many hidden layers as the gradients tend to vanish in deep networks. When should each of the common ...
1
vote
1answer
31 views

What is the equation of the separation line for this neuron with identity activation?

I have a single neuron with 2 inputs, and identity activation, where f is activation function and u is output: $u = f(w_1x_1 + ...
2
votes
1answer
82 views

How to choose the activation function in neuroevolution?

I am developing a NEAT flappy bird game, and it doesn't work, the system stays stupid for 300 generations. I chose tanh() for activation, just because it's included in JS. I can't find a good ...
2
votes
0answers
17 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
2
votes
1answer
208 views

Is a non-linear activation function needed if we perform max-pooling after the convolution layer?

Is there any need to use a non-linear activation function (ReLU, LeakyReLU, Sigmoid, etc.) if the result of the convolution layer is passed through the sliding window max function, like max-pooling, ...
5
votes
1answer
867 views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
1
vote
0answers
27 views

What activation functions are better for what problems?

I’ve been reading about neural network architectures. In certain cases, people say that the sigmoid "more accurately reflects real-life" and, in other cases, functions like hard limits reflect "the ...
6
votes
1answer
277 views

What is the mathematical definition of an activation function?

What is the mathematical definition of an activation function to be used in a neural network? So far I did not find a precise one, summarizing which criterions (e.g. monotonicity, differentiability, ...
2
votes
1answer
97 views

Why is a softmax used rather than dividing each activation by the sum?

Just wondering why a softmax is typically used in practice on outputs of most neural nets rather than just summing the activations and dividing each activation by the sum. I know it's roughly the same ...
3
votes
1answer
536 views

Do all neurons in a layer have the same activation function?

I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++. In my current design ...
2
votes
1answer
40 views

What would be the implications of mistakenly adding bias after the activation function?

I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was ...
3
votes
2answers
75 views

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four ...
2
votes
2answers
294 views

Why is there a sigmoid function in the hidden layer of a neural network? [duplicate]

I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is ...
1
vote
2answers
151 views

When should I use a linear activation instead of ReLU?

I have read this post: How to choose an activation function?. There is enough literature about activation functions, but when should I use a linear activation instead of ReLU? What does the author ...
5
votes
4answers
1k views

Do neurons of a neural network model a linear relationship?

I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics ...
2
votes
2answers
111 views

Can multiple activation functions be replaced with a single activation function?

I'm just started to learn deep learning and I have a question about this neural network: I think $h_1$, $h_j$ and $h_n$ are perceptrons. So, if they are perceptrons, all of them will have an ...
0
votes
1answer
100 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
7
votes
3answers
2k views

Why is the derivative of the activation functions in neural networks important?

I'm new to NN. I am trying to understand some of its foundations. One question that I have is: why the derivative of an activation function is important (not the function itself), and why it's the ...
4
votes
2answers
197 views

Is PReLU superfluous with respect to ReLU?

Why do people use the $PReLU$ activation? $PReLU[x] = ReLU[x] + ReLU[p*x]$ with the parameter $p$ typically being a small negative number. If a fully connected layer is followed by a at least two ...
2
votes
1answer
40 views

Why are activation functions independent layers in CNNs rather than part of convolutional layers?

I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization layers. A common normalization layer is a ReLU layer. But I never encountered an ...
6
votes
1answer
591 views

What's the advantage of log_softmax over softmax?

Previously I have learned that the softmax as the output layer coupled with the log-likelihood cost function (the same as the ...
3
votes
2answers
106 views

What kind of functions can be used as activation functions?

I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an ...
4
votes
1answer
129 views

How do intermediate layers of a trained neural network look like?

Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a ...
2
votes
1answer
218 views

Regarding the output layer's activation function for continuous action space problems

I'm interested in building a (deep) RL agent for solving a continuous problem (which splits something into portions). In all examples I've seen so far, e.g., solving the continuous lunar lander, ...