# Do good approximations produce good gradients?

Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my gradient is a decent approximation of the gradient of the true classifier everywhere?

Specifically, suppose that I have a true loss function, $f$, and an estimation of it, $f_k$. Is it the case that there exists a $c$ (dependent on $f_k$) such that for all $x$ and $\epsilon > 0$ if $|f(x)-f_k(x)|<\epsilon$ then $|\nabla f(x) - \nabla f_k(x)|<c\epsilon$? This isn’t true for general functions, but it may be true for neural nets. If this exact statement isn’t true, is there something along these lines that is? What if we place some restrictions on the NN?

The goal I have in mind is that I’m trying to figure out how to calculate how long I can use a particular sample to estimate the gradient without the error getting too bad. If I am in a context where resampling is costly, it may be worth reusing the same sample many times as long as I’m not making my error too large. My long-term goal is to come up with a bound on how much error I have if I use the same sample $k$ times, which doesn’t seem to be something in the literature as far as I’ve found.

In general $$|f(x) - f_k(x)| \leq \epsilon$$ doesn't ensure $$|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$$. And for neural networks there is no reason to believe it will happen either.
Edit: (Trying to come up with a negative example) Consider $$f(x) = g(x) + \epsilon \sin (\frac {kx} {\epsilon})$$. $$g(x)$$ is some neural network. Now, we train some another neural network $$h(x)$$ to fit $$f(x)$$ and after training we get $$h(x) = g(x)$$ ($$h(x)$$ and $$g(x)$$ have same weights precisely). However, $$\nabla f_x =\nabla g_x + k\cos (\frac {kx} {\epsilon})$$ is not arbitarariy close $$\nabla g_x$$.