6 votes
Accepted

How is the max function differentiable wrt multiple arguments?

Not sure wether this question is suitable here or should rather be on the math SE but here goes. First restrict to two arguments, the general case is similar. Let $f(x,y):=max(x,y)$. This is a ...
quarague's user avatar
  • 281
3 votes
Accepted

Direct formula for calculating the optimum matrix which minimizes the perceptron error

The idea is correct, the last formula is wrong. In general $X$ will not be square, usually one has much more data than parameters. The data points will also be in general position, so that $X$ has ...
Lutz Lehmann's user avatar
2 votes

Direct formula for calculating the optimum matrix which minimizes the perceptron error

As you understand, $E$ is the definition of loss function. This function defines square of the difference between weights applied to $X_i$, namely output of the perception, and $Y_i$ the desired ...
OmG's user avatar
  • 1,816
2 votes
Accepted

Reason for relaxing limit in derivative in this context?

The equation $$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$ is justified from taylor series. It is not derived from the limit definition of a derivative. Let $f'(a)$ and $f''(a)$ exist for $a$ in ...
Taw's user avatar
  • 1,241
1 vote

Reason for relaxing limit in derivative in this context?

It is just an assumption. If we assume $\epsilon$ is small enough (depending on the function $f$), you can remove the limit $\lim_{\epsilon \to 0}$ for the approximation.
OmG's user avatar
  • 1,816
1 vote

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

TL;DR: This has to do with the way A. Ng has defined back propagation for the course. Left Column This is only with respect to one input example and so the $\frac{1}{m}$ factor reduces to 1 and can ...
respectful's user avatar
  • 1,106
1 vote
Accepted

What is the correct partial derivative of $Y^c$ with respect to $A_{ij}^{kc}$?

After some reflection I noticed that the actual final expression should contain derivatives of the alphas w.r.t. $A_{ij}^k$ too, because the alphas cannot be constants that do not depend on $A_{ij}^k$....
mlerma54's user avatar
  • 141
1 vote

What is the rigorous and formal definition for the direction pointed by a gradient?

If $u$ is a vector, the direction pointed by the vector is defined as $\dfrac{u}{\lVert {u}\rVert}$ where $\lVert \cdot \rVert$ is the 2 norm (euclidean norm).
Taw's user avatar
  • 1,241
1 vote

What does it mean "having Lipschitz continuous derivatives"?

Consider a function $f(x) : \mathcal{R}^m\rightarrow\mathcal{R}^n$ defined for $x \in X$. If $f$ is Lipschitz continuous, it has three main properties: $f(x)$ is continuous for all $x \in X$ $\frac{d ...
Taw's user avatar
  • 1,241
1 vote

Why does critical points and stationary points are used interchangeably?

From reading the text, it's clear that the authors are using critical point to mean the same thing as stationary point, so they are not using the proper mathematical definition. More generally, ...
Taw's user avatar
  • 1,241
1 vote

Why does critical points and stationary points are used interchangeably?

A critical point of a function $f$ can be a stationary point (i.e. $f'(x) = 0$), or a point where the derivative is undefined (for example, in the case of the absolute value function $f(x)$, $x=0$ is ...
nbro's user avatar
  • 40.1k

Only top scored, non community-wiki answers of a minimum length are eligible