2
$\begingroup$

I have CNN model to classify 2 classes. (Yes or No) I use categorical_crossentropy loss and softmax activation at the end. For input I use image with all 3 channels, for output I use One hot encoded vector ([0,1] or [1,0])

I have function that guaranty me, that each batch I have same number of one and another class, so the classes are not unevenly represented.

What happened when I train the model is that I am stuck at same loss while trening,...

I assume that model predict always same class and half in batch has loss 0 half of them max, so that bring it to 8 all the time,...

What could went wrong?

The network is something like this :

x = Conv2D(16, (3, 3), padding='same')(input_img)
x = LeakyReLU(0.1)(x)
x = Conv2D(32 , (3, 3), padding='same')(x)
x = LeakyReLU(0.1)(x)
x = MaxPooling2D((2, 2))(x)
x = Dropout(0.25)(x)
x = Conv2D(32 , (3, 3), padding='same')(x)
x = LeakyReLU(0.1)(x)
x = Conv2D(48 , (3, 3), padding='same')(x)
x = LeakyReLU(0.1)(x)
x = MaxPooling2D((2, 2))(x)
x = Dropout(0.25)(x)

x = Flatten()(x)
x = Dense(4096)(x)
x = LeakyReLU(0.1)(x)
x = Dropout(0.5)(x)
x = Dense(2048)(x)
x = LeakyReLU(0.1)(x)
x = Dropout(0.5)(x)
out = Dense(2, activation='softmax', name='table')(x)

model = Model(input_img, out)
model.compile(optimizer='adam', loss= 'categorical_crossentropy')

Training Loss:

enter image description here

$\endgroup$
  • $\begingroup$ Sorry, that you're question is waiting for 2 days without providing an answer. As an excuse, i upvoted the post, so your reputation will grow. $\endgroup$ – Manuel Rodriguez Sep 29 at 14:44
  • $\begingroup$ I figure out, that sometimes bigger batch size helps : from 32 to 256,... But I would appreciate some theoretical and practical advice to solve this king of errors... $\endgroup$ – Marko Zadravec Sep 29 at 20:13
  • $\begingroup$ There are many things that you could be doing wrong. 1. Are you sure your initialisation is correct? An unchanging error is sometimes a sign that an entire layer is dead, hence no ability to change the error. 2. Are you sure your backprop is updating everything? 3. Are you sure that you are definitely using a one hot vector as the output 4. If you try training your CNN on something else, is it able to learn? 5. Make sure your cross entropy loss function is correct. I can almost guarantee a behaviour like this is caused by an error in the final 2 or so layers of the network. $\endgroup$ – Recessive Sep 30 at 6:00
  • $\begingroup$ @Recessive : 1. Initialisation is handle by Adam,.. and I use LeakyReLU not to produce 0 output 2. Yes 3. Yes I always use [0,1] or [1,0]. 4. I need to check this, but I think so (Usually if I increase batch size it train better, but I need to increase it to 256 or more,...). 5. I use default Keras loss function implementation. Regarding last layers : They are normal dense layers with LeakyReLU activations. What did you have in mind as a potential error? $\endgroup$ – Marko Zadravec Oct 1 at 5:18
  • 1
    $\begingroup$ Is this the training loss or validation loss ? Did you try to print out the predictions made by your network ? $\endgroup$ – Joseph Budin Oct 1 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.