# How should the values of the filters of a CNN change?

I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation. But one thing I noticed was the "just positive" derivative of the ReLU function.

My network structure is the following:

• (Input 28x28)
• Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
• Max Pool Layer (Size 2x2) with RELU
• Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
• Max Pool Layer (Size 2x2) with RELU
• Dense (128)
• Dense (10)

I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?

Whenever I look at any of the filter's values, they decreased after training. Is that correct?

By the way, I am using stochastic gradient descent with a fixed learning rate for training.

• By negative or positive direction do you mean that they respectively decrease or increase in value? – nbro Apr 22 at 12:13
• Yes that is what I meant. I'll update my post. – jutu OOtv Apr 23 at 1:34

The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $$-\infty$$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $$-\infty$$. You will have to increase one or more weights.