Hot answers tagged

4

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of samples. In the second case you have multiple classes and in the current form it's only for a single sample. In the first case there is only one output, if you ...


4

Short answer: Generally, you don't need to do softmax if you don't need probabilities. And using raw logits leads to more numerically stable code. Long answer: First of all, the inputs of the softmax layer are called logits. During evaluation, if you are only interested in the highest-probability class, then you can do argmax(vec) on the logits. If you want ...


4

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (1989, NIPS) by John S. Bridle. As a side note, the softmax function (with base $b = e^{-\beta}$) $$\sigma (\mathbf {z} )_{i}={\frac {e^{-\beta z_{i}}}{\sum _{j=...


3

When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus. If you have a function $$f:\mathbb{R}\rightarrow\mathbb{R}$$ the derivative is a function on its own and you have $$f':\mathbb{R}\rightarrow\mathbb{R}.$$ If you ...


2

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low network outputs, that can be translated as "don't know what to do". In other words, the network can result in: "I don't know what to do now&...


1

You can find a description of this distribution (which is also known as categorical distribution, which you probably already heard of) in section 2.3.2 (p. 35) of the book Machine Learning: A Probabilistic Perspective (by K. Murphy). You can also find there and in the previous section a description of the related Bernoulli, binomial and multinomial (the most ...


1

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE). Here's the BCE (equation 4.90 from this book) $$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$ where $t_{n} \in\{0,1\}$ is the target $y_n \in [0, 1]$ is the prediction (as produced by ...


1

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation: $$ \sigma(\mathbf z)_{i} = \frac{e^{z_i}}{\sum_k e^{z_k}} $$ I think, one of the most effective ways of not to get confused about all these matrices with different ...


1

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when ...


1

I know this is not a straight answer to your question, but I couldn't comment on your post so decided to post it (so maybe I will delete it after you received a better answer). I think this playlist by sentdex can be handy as he goes through a lot of details to teach a neural network model that can drive cars in GTA-V by simply looking at each frame of the ...


1

It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a softmax + crossentropy module all-in-one, then it becomes numerically stable (y-p) Slides from ...


Only top voted, non community-wiki answers of a minimum length are eligible