# Tag Info

4

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network to "get UNLUCKY and get knocked into a BAD sectors of parameter space corresponding to a sudden decrease in accuracy -- sometimes it can recover from this ...

4

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too). You may require the output of the neural network to be a ...

3

Yes, this is actually a limitation known as catastrophic forgetting. A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on the weights important for those tasks". See Overcoming catastrophic forgetting in neural networks for details. Another approach is Learning without forgetting. ...

2

Derivative of the sigmoid curve is 0 when the output is 0 or 1 as you can see from the image above. The technique you are referring to is called label-smoothing which is used in various applications (e.g. GANs) but I can see how it would be applicable here also, by helping to avoid 0-gradients saturating learning. To answer your third question, sigmoids are ...

2

There are a variety of possible things that could be wrong, but let me give you some potentially useful information. Neural networks with ReLU activation functions are Turing complete for a computation with on order as many steps as the network contains nodes - for a recurrent network (an RNN), that means the same level of turing completeness as any finite ...

2

The general answer to the behavior of combining common activation functions is that the laws of calculus must be applied, specifically differential calculus, the results must be obtained through experiment to be sure of the qualities of the assembled function, and the additional complexity is likely to increase computation time. The exception to such ...

2

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the weight vector needs more update to be trained properly and the number of epochs needed for the network to get ...

2

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low network outputs, that can be translated as "don't know what to do". In other words, the network can result in: "I don't know what to do now&...

2

I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this). They don't in general. In the quoted text, there is an explicit constraint that means this can be the case: If it is desirable to predict a probability of a binary class (emphasis mine). This means that the target value $y \in \{... 1 It is not the sigmoid in particular. LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget). Observe the top horizontal line in the image below. The line represents the ... 1 I know this is not a straight answer to your question, but I couldn't comment on your post so decided to post it (so maybe I will delete it after you received a better answer). I think this playlist by sentdex can be handy as he goes through a lot of details to teach a neural network model that can drive cars in GTA-V by simply looking at each frame of the ... 1 Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE). Here's the BCE (equation 4.90 from this book) $$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$ where$t_{n} \in\{0,1\}$is the target$y_n \in [0, 1]$is the prediction (as produced by ... 1 I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to apply a scaling. In any case I would consider what the target distribution is before using a sigmoid and applying a scaling. The thing about a sigmoid is that the ... 1 There are several functions that can be denoted as sigmoid functions, such as the logistic function and the hyperbolic tangent, given that they have an$S\$-shaped curve. You can find more info about them in the related Wikipedia article. However, when people use the term sigmoid function, they typically refer to the logistic function, which is a function of ...

1

It seems like you're suffering from the the dying ReLU problem. ReLU enforces positive values so the weights and biases your network learned are leading to a negative value passed through the ReLU function - meaning you would get 0. There are a few things you can do. I do not know the exact format of your data, but if it is MNIST it is possible you simply ...

1

I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing. The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and ...

1

It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input. This is a trivial fact that applies to a single layer perceptron. This is true for for the single layer case for any activation function, also a trivial fact. When the layer number is between ...

1

While I have not determined if there are problems that cannot be solved with ReLU, I have found ample documentation in the literature that XOR is solvable with as few as 1 hidden node. The solution is simpler than I thought. The output layer needs connections, not just to the intermediate layer, but directly to the input layer as well. This allows the ...

Only top voted, non community-wiki answers of a minimum length are eligible