3
$\begingroup$

I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them has a higher variance and thus a slower convergence rate. I want to mathematically prove this, but I don't know where to start. Is there a standard reference in statistics that I can consult for example proofs of convergence rates in ML/statistics?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The Rate of Convergence of an Estimator will depend on a number of factors including:

  • the type of estimator used,
  • the data set used for training,
  • and the tuning parameters of the estimator.

However, in general, one can expect that an estimator will converge at a rate that is at least as fast as the inverse of the number of training examples used.

...I suspect one of them has a higher variance and thus a slower convergence rate... Yes, if two estimators are sampling from the same distribution, the estimator with higher variance will converge more slowly than the estimator with lower variance &, this can be important when you are choosing between different algorithms to use for training your models, but There is no general rule; the choice depends on the specific situation, the one with the faster convergence rate will generally be more accurate that obvious.

"the rate of convergence provide useful insights when using iterative methods for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the asymptotic behavior of a sequence does not give conclusive information about any finite part of the sequence."

There are a few different ways to approach this. One way would be to use the law of large numbers(~4). This states that, as the number of samples increases, the average of the sample values will converge to the true value of the quantity being estimated.

Another way would be to use the Central Limit Theorem(~5). This states that, as the number of samples increases, the distribution of the sample values will become more and more like a normal distribution. The mean of this distribution will be the true value of the quantity being estimated.

Here are some references may be relevant:

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .