Hot answers tagged


You need 10-bits ($2^{10} = 1024$) to represent 1000 classes.


Both the sparse categorical cross-entropy (SCE) and the categorical cross-entropy (CCE) can be greater than $1$. By the way, they are the same exact loss function: the only difference is really the implementation, where the SCE assumes that the labels (or classes) are given as integers, while the CCE assumes that the labels are given as one-hot vectors. Here ...


It's the loss function. I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why: From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But ...


It takes at least 10 bits to represent any number between $1-1000$ because $2^{10} = 1024$. This means that if one was trying to represent 1 of the 1000 classes, one would need at least 10 bits. However, having these 10 bits set correctly for each input is really hard and would require overfitting to ensure it.

Only top voted, non community-wiki answers of a minimum length are eligible