I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted.
What I don't understand is how MSE a positive only loss value can train a back propagation network. Positive only losses result in weights only incrementing in their pre-initialised sign (- or +) thus they just explode or vanish.
Positive only losses result in weights only incrementing in their pre-initialised sign
No.
In gradient descent, you use the (partial) derivative or the gradient (if you're using vector operations and you have more than one parameter/weight) of the loss function, not the loss function.
So, let's say you have the function $f(x; w) = wx$ that you want to optimize wrt to $w$. Suppose $y$ is the ground-truth label and we use the mean squared error to optimise this function, i.e. $L(x; y; w) = (w x - y)^2$, then the gradient descent step is
$$w \leftarrow w - \alpha \frac{d L}{dw}$$
Now, what is the its derivative of $L$ wrt to $w$, i.e. $\frac{d L}{dw}$? It's
$$2x(w x - y)$$
Now, let's say $x = 1$, $w = 1$ and $y = 2$, then you have that the derivative is
$$2*1(1*1 - 2) = 2 * (-1) = -2$$
Now, let's say $x = 2$, $w = 1$ and $y = 1$, then you have that the derivative is
$$2*2(1 *2 - 1) = 4$$
So, clearly, the derivative can be positive or negative, even if the loss function is always positive. Note that I've used the same $w$ in both examples and the derivatives have different signs.
