How should the values of the filters of a CNN change?

Question

I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation. But one thing I noticed was the "just positive" derivative of the ReLU function.

My network structure is the following:

• (Input 28x28)
• Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
• Max Pool Layer (Size 2x2) with RELU
• Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
• Max Pool Layer (Size 2x2) with RELU
• Dense (128)
• Dense (10)

I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?

Whenever I look at any of the filter's values, they decreased after training. Is that correct?

By the way, I am using stochastic gradient descent with a fixed learning rate for training.

Answer 1

All weight matrices in a Neural Network, adapt to map input and output. ReLU, as you pointed out, doesn't give negative derivatives. You're right. But notice the weight update equation in Backpropagation, it uses a multitude of parameters like:

• Error value
• Activation from the previous layer
• Weight matrix of the previous layer
• The pre-activation cache of the previous layer

These can be positive or negative. Hence weight updations dW and db can be +ve or -ve, allowing for decreasing (or increasing) value of weights.

Answer 2

The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $-\infty$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $-\infty$. You will have to increase one or more weights.