How much extra information can we conclude from a neural network output values?

Question

Consider I have a 3 layers neural network.

• Input Layer containing 784 neurons.
• Hidden layer containing 100 neurons.
• Output layer containing 10 neurons.

My objective is to make an OCR and I used MNIST data to train my network.

Suppose I gave the network an input taken from an image, and the values from the output neurons are the next:

• $0: 0.0001$
• $1: 0.0001$
• $2: 0.0001$
• $3: 0.1015$
• $4: 0.0001$
• $5: 0.0002$
• $6: 0.0001$
• $7: 0.0009$
• $8: 0.001$
• $9: 0.051$

When the network returns this output, my program will tell me that he identified the image as number 3.

Now by looking at the values, even though the network recognized the image as 3, the output value of number 3 was actually very low: $0.1015$. I am saying very low, because usually the highest value of the classified index is as close as 1.0, so we get the value as 0.99xxx.

May I assume that the network failed to classify the image, or may I say that the network classified the image as 3, but due to the low value, the network is not certain?

Am I right thinking like this, or did I misunderstand how does the output actually works?

Answer 1

From the values you have provided I can easily guess your output layer has the sigmoid (do clarify!) activation function. For sigmoid activation function this can be a quite normal occurrence. Also maybe the number of training epochs is not high enough.

The case you have mentioned of 0.99 is generally in the case of the output being subjected to a softmax probability function. Although 0.99 is still achievable using sigmoid activation, it will depend on your network hyper-parameters and in general the data. If data is very easily separable the sigmoid will generally give very high contrasting difference among classes. Also if the hidden network has ReLu it becomes easy to provide contrasting class scores due to its huge scale difference compared to a sigmoid activation in the hidden layer.

The point here to note is contrasting difference among classes, because your sigmoid might give 0.99 for the correct class while 0.9 for other classes, which is undesirable.