Hot answers tagged

3 votes
Accepted

Why cannot linear activation functions be used to approximate any function?

However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is ...
user avatar
  • 671
2 votes
Accepted

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

A network with ReLU activation can predict negative values; we put ReLU between the hidden layers but return the output of the final layer without any activation function, or with a linear activation ...
user avatar
2 votes

Why should one ever use ReLU instead of PReLU?

I suppose, the situation is as follows - PReLU increases the expressiveness of a model for a bit at a small cost, but the gain is almost negligible as well (...
user avatar
1 vote

Why and when do we use ReLU over tanh activation function?

For a discussion about the advantages of ReLU, see the original paper by Glorot (2011) "Deep sparse rectifier neural networks". "Efficient Backprop" is a 1998 paper. At the time ...
user avatar
  • 185
1 vote
Accepted

Where does the "rectified" in ReLU come from?

I think it is by analogy with an electrical rectifier. A rectifier allows current to flow in one direction but blocks current in the other direction. Or if you prefer it allows voltage in one polarity ...
user avatar
1 vote
Accepted

What is meant by non-linearity in Convolutional Neural Networks? And why do we focus on removing it entirely?

The concept of non-linearity is not only restricted to Convolutional Networks but can be seen in RNNs, and any feed forward neural networks. Without a non-linear activation function, two feed forward ...
user avatar
  • 126
1 vote
Accepted

Are any non-injective activation functions used?

There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$. ...This suggests that Swish can be loosely viewed as a smooth function which nonlinearly interpolates between ...
user avatar
  • 345
1 vote

Effects of ReLU Activation on Convexity of Loss Functions

You're missing a couple of quite important concepts: Universal approximation theorem: with enough parameters a neural network can approximate any function. Basically every loss function is non convex....
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible