# Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $$A, W$$.

As mentioned in the regularization notebook (see below) in order to do derivation of regularized $$J$$ (cost function), the changes only concern $$dW^{}$$, $$dW^{}$$ and $$dW^{}$$. For each, you have to add the regularization term's gradient.(No impact on $$dA^{}$$, $$db^{}$$, $$dA^{}$$ and $$db^{}$$ ?) But I am doing it using the chain rule then I am getting change in values for $$dA^{}$$ , $$dZ^{}$$, $$dA^{}$$, $$dW^{}$$ and $$db^{}$$.

Please refer below how I calculated this ? Can someone explain why I am getting different results?

What is the derivative of L2 reguarlization w.r.t $$dA^{}$$ ? (in equation 1)

So my questions are

1) Derivative of L2 regularization cost w.r.t $$dA^{}$$

2) How adding regularization term not affecting $$dA^{}$$, $$db^{}$$, $$dA^{}$$ and $$db^{}$$ (i.e. $$dA$$ and $$db$$) but changes $$dW$$'s ?

• Hi. Please, see this https://ai.meta.stackexchange.com/q/1654/2444. – nbro Mar 21 at 19:08
• Thanks for the suggestion. Made title more clear now :) – user110244 Mar 22 at 4:34
• Its a good thing you are trying to derive the equations yourself, but it is quite difficult for someone to debug your equations for you. So I would suggest you to try to find the mistake yourself, it'll prepare you for future mistakes you are going to make. As for your question the $L2$ is w.r.t weights and hence impact is directly on weights, you don't have to go through chain rule of activations to reach a weight. All get the same weightage in the L2 term. A better way to derive this would be just to forget the cross entropy term and make your cost just L2. – DuttaA Mar 24 at 9:30
• This will result in a cost dependent linearly only on square of weights, whose derivative w.r.t weights you can easily take....there is no interfering terms of $dA$ and other stuff. The cost function $J_{reg}$ is modularized/independent. So try to find the derivatives of cross entropy and L2 individually like I suggested above and just add them. This is the most cited tutorial (as far as I have seen) for backprop: mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example – DuttaA Mar 24 at 9:32
• @DuttaA One of my question is why we need to avoid chain rule in case of derivative of regularised cost function especially for regularization part ? I know if I separate cost function into two part where part_1 is cross_entropy and part_two is l2_regularization cost and then calculate derivative where for part_1 we can use chain rule and for part_2 we don't need to use chain rule as this makes reglarization impact on dW zero then we can easily arrive at the derivative. – user110244 Mar 24 at 9:46