I am trying to better understand the Least Mean Squares algorithm, in order to implement it programmatically.
If we consider its weight updating formula $$w(n+1) = w(n) + \rho * \text{error}(i)x(i),$$ where $w(n + 1)$ is the new weight of the classifier function, $w(n)$ is its current weight and $x(i)$ is the $i$th element of a training dataset, what should $\rho$ be?
From what I have found online, $ρ$ is supposed to be $0 < \rho < \frac{2}{trace(X^TX)}$, where $X$ is a matrix with all the training data the algorithm has processed at that point. One idea that I had, was to take $\rho = \frac{1}{trace(X^TX)} < \frac{2}{trace(X^TX)}$, but I do not know if that is correct. Also, one characteristic that this value has is that it changes with each iteration of the algorithm, as more samples are added to matrix $X$.
So, what is a good value for $\rho$? Should it change during the execution of the algorithm or should it stay the same?
