0
$\begingroup$

I learned that when CNN filters are defined, they are initialized with random weights and bias(Im not sure about bias).

Then as learning step goes on, the weight values change and each filter makes its own feature map.

What I don't understand is that, if filter is initialized with random values, is there any chance that

  1. different filters make the same feature map
  2. feature map varies every time it repeats.

It seems little unefficient to initialize the weights of every filter randomly. More precisely, I think (in most of the networks) the number of filters is too small to get meaningful features.

Is the second case is why CNN has randomness?

$\endgroup$
2
  • $\begingroup$ first of all its is best to initialize randomly according to some distribution, for instance if you are using Relu activation, initialize according He-normal, or He-uniform, see the corresponding papers for why. second, if you initialize all weights similarly well then they all be treated in the same way by the gradient descent, and converge to similar values. finally in stochastic optimization, you need some kind to diversity to be able to explore the search space and find the solution. You do not no where the optimum solution is you need to find it, diversity ensures that you have a chance. $\endgroup$
    – Reza_va
    Commented Jan 23 at 11:06
  • $\begingroup$ Can you please put your specific question in the title? Thanks. $\endgroup$
    – nbro
    Commented Jan 26 at 11:10

1 Answer 1

1
$\begingroup$

It is usual to initialize parameters with "a good guess", when you have prior information, in order to help the model converging.

However in deep learning most of the time you have no clue what the weights should be, so you initialize them randomly following certain schedules like Xavier initialization or Kaiming, that prevent gradients from vanishing or exploding (due to a bad initialization) by setting the mean activations to be zero and the variance of the activations to be constant accross all layers.

Regarding your questions :

  1. Yes it is totally possible that multiples filters learn the same thing. In fact, a good part of the filters may be completly useless, which is the idea behing network pruning that tries to delete neurons in a network while keeping the same performances.

  2. Yes, the network you'll obtain after training will depend on the initialization. You can have different results depending on the initialization (just permuting the filters, for instance, give a new network with same capabilities), since anyway you will never get the global optimum for your task but only sub-optimal weights. However if trained properly, the final results should not very too much.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .