So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ReLUs) and the last output layer will be linear because we are trying to catch a specific number and not a probability. I've done the forward propagation calculations but I'm stuck at the back propagation ones.
Let's say that I'm gonna use the cross entropy loss function: (we will implement the MSE as well) $-(y \log (a)+(1-y) \log (1-a))$. (I understand that this is not a good option for a regression problem)
So we can easily find the dJ/dA $d A=\frac{\partial J}{\partial A}=-\left(\frac{y}{A}-\frac{1-y}{1-A}\right)$ of the last layer and we can start going backwards finding the $\frac{\partial J}{\partial Z^{[L]}}$ which we can calculate from the equation: $d Z^{[L]}=d A*g^{\prime}\left(Z^{[L]}\right)$ The problem lies at the second part of this equation where the derivative of g is.
What will be the outcome since we have a linear activation function which derivative is equal with 1? (Activation function: f(x) = x, f'(x) = 1)
Will it be an identity matrix with the shape of Z[L] or a matrix full of ones with the same shape again? I'm asking about the term $g^{\prime}\left(Z^{[L]}\right)$.
Many thanks.
$signs e.g.$\nabla_{Z^{[L]}} = \frac{\partial J}{\partial Z^{[L]}}$becomes $\nabla_{Z^{[L]}} J = \frac{\partial J}{\partial Z^{[L]}}$. I suggest you do this in the question to make it easier to read. Use edit link to alter the question. Also, can you be more specific about what "the second part of this equation" refers to - say which term you mean? Also be clear about "outcome" - are you asking about the overall loss value, or just the value of the term yo uare having trouble with? $\endgroup$ – Neil Slater Oct 13 '20 at 10:17