# Is there anything that ensures that convolutional filters don't end up the same?

I trained a simple model to recognize handwritten numbers from the mnist dataset. Here it is:

model = Sequential([
Conv2D(filters=1, kernel_size=(3,1), padding='valid', strides=1, input_shape=(28, 28, 1)),
Flatten(),
Dense(10, activation='softmax')])


I experimented with varying the number of filters for the convolutional layer, while keeping other parameters constant(learning rate=0.0001, number of episodes=2000, training batch size=512). I used 1, 2, 4, 8, and 16 filters, and the model accuracy was 92-93% for each of them.

From my understanding, during the training the filters may learn to recognize various types of edges in the image (e.g, vertical, horizontal, round). This experiment made me wonder whether any of the filters end up being duplicate -- having the same or similar weights. Is there anything that prevents them from that?

• Since you've edited the title of this, it's changed into being the same as my follow up question. Not sure what admins will do with it, but keep an eye on it in case. Thanks for the great question! Dec 30 '20 at 16:18
• Hey, I did just change the title -- I just revisited the question and I realized I've made a mistake by not including the "don't". This might have been confusing, since in the description I ask "whether any of the filters end up being duplicate" Dec 30 '20 at 16:26
• Yeah makes sense. Let's see if we can get an interesting response between our questions then. Dec 30 '20 at 16:32

No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has $$n$$ features, but 2 of them are the same, then the model effectively has $$n-1$$ features, which is a less expressive model than that of $$n$$ features, and therefore usually has a larger loss function.