So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ReLUs) and the last output layer will be linear because we are trying to catch a specific number and not a probability. I've done the forward propagation calculations but I'm stuck at the back propagation ones.

Let's say that I'm gonna use the cross entropy loss function: (we will implement the MSE as well) $-(y \log (a)+(1-y) \log (1-a))$. (I understand that this is not a good option for a regression problem)

So we can easily find the dJ/dA $d A=\frac{\partial J}{\partial A}=-\left(\frac{y}{A}-\frac{1-y}{1-A}\right)$ of the last layer and we can start going backwards finding the $\frac{\partial J}{\partial Z^{[L]}}$ which we can calculate from the equation: $d Z^{[L]}=d A*g^{\prime}\left(Z^{[L]}\right)$ The problem lies at the second part of this equation where the derivative of g is.

What will be the outcome since we have a linear activation function which derivative is equal with 1? (Activation function: f(x) = x, f'(x) = 1)

Will it be an identity matrix with the shape of Z[L] or a matrix full of ones with the same shape again? I'm asking about the term $g^{\prime}\left(Z^{[L]}\right)$.

Many thanks.

  • $\begingroup$ You don't want to implement cross entropy loss with linear activation. The cross entropy loss function is poorly defined when inputs are less than 0 or greated than 1. It can be done though, so if you are still interested in the theory, can you please specify in more detail what the loss function is - give a formula, because there are a couple of variations that are called "cross entropy". $\endgroup$ – Neil Slater Oct 13 '20 at 10:00
  • $\begingroup$ You can use LaTex between $ signs e.g. $\nabla_{Z^{[L]}} = \frac{\partial J}{\partial Z^{[L]}}$ becomes $\nabla_{Z^{[L]}} J = \frac{\partial J}{\partial Z^{[L]}}$. I suggest you do this in the question to make it easier to read. Use edit link to alter the question. Also, can you be more specific about what "the second part of this equation" refers to - say which term you mean? Also be clear about "outcome" - are you asking about the overall loss value, or just the value of the term yo uare having trouble with? $\endgroup$ – Neil Slater Oct 13 '20 at 10:17
  • $\begingroup$ I just edited the post. You can have a second read now. Many thanks. $\endgroup$ – ChrisP Oct 13 '20 at 15:07
  • 1
    $\begingroup$ Can you please put your main question in the title? Right now, your title is not a question and it's not very descriptive. $\endgroup$ – nbro Oct 13 '20 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.