2
$\begingroup$

I'm training two CNNs (AlexNet e GoogLeNet) in two differents DL libraries (Caffe e Tensorflow). The networks was implemented by dev teams of each libraries (here and here)

I reduced the original Imagenet dataset to 1024 images of 1 category -- but setted 1000 categories to classify on the networks.

So I trained the CNNs, varying processing unit (CPU/GPU) and batches sizes, and I observed that the losses converges fastly to near zero (in mostly times before 1 epoch be completed), like in this graph (Alexnet on Tensorflow):

In portuguese, 'Épocas' is epochs and 'Perda' is loss.

The weight decays and initial learning rate are the same as used on models that I downloaded, I only changed the dataset and the batch sizes.

Why my networks are converging this way, and not like this way?

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean, you settled 1000 categories? $\endgroup$
    – BlueMoon93
    Commented Dec 5, 2017 at 10:31
  • $\begingroup$ I specified to network/library 1000 different output categories, but in practice I just give 1 to the network. I dont know exactly how Tensorflow/Caffe works, but I think that this value (1000) is used in softmax layer $\endgroup$
    – 648trindade
    Commented Dec 5, 2017 at 17:47

1 Answer 1

Reset to default
2
$\begingroup$

Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph.

Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thats make a lot of sense. But why loss converges faster if my batch is too small? With sizes lower than 32, the networks converges with just 10% of an epoch (aprox. 100 images) $\endgroup$
    – 648trindade
    Commented Dec 5, 2017 at 18:10
  • $\begingroup$ You have less variance when you have a smaller batch size, so the network ignores the noise even faster $\endgroup$
    – BlueMoon93
    Commented Dec 5, 2017 at 18:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .