Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

18 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
43 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
2
votes
0answers
29 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
2
votes
1answer
45 views

Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?

The associative property of multidimensional discrete convolution says that: $$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$ where $h_1$ and $h_2$ are the filters and $x$ is ...
2
votes
0answers
36 views

What work has been done with Poisson-style regression via neural networks with exponential activation function?

The first neural net I wrote was a classifier. After that, I learned that neural nets can be used for regression tasks, even quantile regression. It has become clear to me that the usual games with ...
2
votes
0answers
43 views

Which activation functions should I use for polynomial regression?

I am a beginner in machine learning and neural networks. I have only used neural networks for classification problems. My aim is to modify it so that it can work for polynomial regression as well. In ...
2
votes
0answers
29 views

Single label classification into hierarchical categories using a neural network

I am working on a classification problem into progressive classes. In other words, there is some hierarchy of categories in such a way, that A < B < C, e.g. low, medium, high, very high. What ...
2
votes
1answer
81 views

How to choose the activation function in neuroevolution?

I am developing a NEAT flappy bird game, and it doesn't work, the system stays stupid for 300 generations. I chose tanh() for activation, just because it's included in JS. I can't find a good ...
2
votes
0answers
17 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
2
votes
0answers
125 views

Non-monotonic activation function and XOR problem with perceptron

I already post the question on Stackoverflow : https://stackoverflow.com/questions/53785922/is-the-use-of-a-non-monotonic-activation-function-is-a-correct-and-viable-solut?noredirect=1#...
1
vote
1answer
53 views

What is the exact structure within the nodes of a hidden layer?

I've been reading on neural networks, but for me, seems like the easiest way for me to learn is seeing some code. I am curious about what is the exact structure within a node of a hidden layer and ...
1
vote
0answers
31 views

If the output of a model is a ridge function, what should the activation functions at all the nodes be?

I have the following assignment. I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped. I ...
1
vote
0answers
32 views

Can SqueezeNet be used for regression?

I want a model that outputs the pixel coordinates of the tip of my forefinger, and whether it's touching something or not. Those would be 3 output neurons: 2 for the X-Y coordinates and 1, with a ...
1
vote
0answers
44 views

Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
1
vote
0answers
40 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
0answers
32 views

What are the pros and cons of the common activation functions?

I have heard that sigmoid activation functions should not be used on neural networks with many hidden layers as the gradients tend to vanish in deep networks. When should each of the common ...
1
vote
0answers
27 views

What activation functions are better for what problems?

I’ve been reading about neural network architectures. In certain cases, people say that the sigmoid "more accurately reflects real-life" and, in other cases, functions like hard limits reflect "the ...
1
vote
0answers
30 views

Activation Function of the Last Layer

I was wondering whether there is a method (not a table of recommendations) that could tell me what activation function to choose if the outputs of the neural network have some interpretation . For ...
0
votes
1answer
60 views

Why is non-linearity desirable in a neural network?

Why is non-linearity desirable in a neural network? I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear ...