While computing error (training phase) its just like any other layer. Error can be negative, hence the ReLU, I guess.

Ah I got that! When m, n, p become equal that is similar to O(cubic). We're multiplying linear vectors, so it makes sense to be the way you showed it. Thanks. Solved.

Yes, it is a function of the number of layers. So assuming $O(n^3)$ for 1 matrix product, it scales for L layers in the same manner, that's how I'm seeing it...

From a different perspective, if we see Neural Networks as plain matrix products (post training feedforward), shouldn't it be something of the form $O(n^3)$? Just my two cents...

Can you show what error you're facing? Does the model compile? I mean, does it even 'start' training, or do you face matrix dimensionality issues?

Yeah, but they do seem promising

I read in one of Andrew's courses, that local optima usually don't look like hills with multiple zero-derivative points; instead GD usually faces 'saddle points', and in general, they're easy to deal with. What's your view on that?