Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..

Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function is denoted as $f'(x)$ or as $\dfrac{dy}{dx}$. The derivative $f'(x)$ gives the slope of $f(x)$ at the point $x$. In other words, it specifies how to scale a small change in the input to obtain the corresponding change in the output: $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.

I have doubt in the equation $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$ given in the paragraph.

In strict sense, the derivative function $f'$ of a real valued function $f$ is defined as

$$f'(x) = \lim_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$

wherever the limit exists.

If I replace the original definition of the derivative as follows

$$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$

then I can obtain the equation given in the paragraph i.e, $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.

But, my doubt is that how can I modify the definition with $\lim\limits_{\epsilon \rightarrow 0}$ to an approximation with out limit? How can the following two are same?

$$f'(x) = \lim_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$


The equation $$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$ is justified from taylor series. It is not derived from the limit definition of a derivative. Let $f'(a)$ and $f''(a)$ exist for $a$ in the interval $(x, x+\epsilon)$. Then, \begin{align*} f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b) \end{align*} where $b$ is some number between $(x, x + \epsilon)$. If $\epsilon << 1$, then $\frac{\epsilon^2}{2}f''(b)$ is small so it can be ignored, leading to the approximation.

taylor's theorem


It is just an assumption. If we assume $\epsilon$ is small enough (depending on the function $f$), you can remove the limit $\lim_{\epsilon \to 0}$ for the approximation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.