0
$\begingroup$

I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation. But one thing I noticed was the "just positive" derivative of the ReLU function.

My network structure is the following:

  • (Input 28x28)
  • Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
  • Max Pool Layer (Size 2x2) with RELU
  • Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
  • Max Pool Layer (Size 2x2) with RELU
  • Dense (128)
  • Dense (10)

I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?

Whenever I look at any of the filter's values, they decreased after training. Is that correct?

By the way, I am using stochastic gradient descent with a fixed learning rate for training.

$\endgroup$
  • 1
    $\begingroup$ By negative or positive direction do you mean that they respectively decrease or increase in value? $\endgroup$ – nbro Apr 22 at 12:13
  • $\begingroup$ Yes that is what I meant. I'll update my post. $\endgroup$ – jutu OOtv Apr 23 at 1:34
0
$\begingroup$

All weight matrices in a Neural Network, adapt to map input and output. ReLU, as you pointed out, doesn't give negative derivatives. You're right. But notice the weight update equation in Backpropagation, it uses a multitude of parameters like:

  • Error value
  • Activation from the previous layer
  • Weight matrix of the previous layer
  • The pre-activation cache of the previous layer

These can be positive or negative. Hence weight updations dW and db can be +ve or -ve, allowing for decreasing (or increasing) value of weights.

$\endgroup$
  • $\begingroup$ Thanks for pointing that out :) $\endgroup$ – jutu OOtv Apr 29 at 21:37
0
$\begingroup$

The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $-\infty$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $-\infty$. You will have to increase one or more weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.