What should the value of $ρ$ in the $w(n+1) = w(n) + \rho*\text{error}(i)x(i)$ formula of Least Mean Squares be? - Artificial Intelligence Stack Exchange most recent 30 from ai.stackexchange.com 2022-01-27T23:11:15Z https://ai.stackexchange.com/feeds/question/31531 https://creativecommons.org/licenses/by-sa/4.0/rdf https://ai.stackexchange.com/q/31531 0 What should the value of $ρ$ in the $w(n+1) = w(n) + \rho*\text{error}(i)x(i)$ formula of Least Mean Squares be? User9123 https://ai.stackexchange.com/users/49544 2021-09-03T20:32:46Z 2021-09-06T11:15:06Z <p>I am trying to better understand the Least Mean Squares algorithm, in order to implement it programmatically.</p> <p>If we consider its weight updating formula <span class="math-container">$$w(n+1) = w(n) + \rho * \text{error}(i)x(i),$$</span> where <span class="math-container">$w(n + 1)$</span> is the new weight of the classifier function, <span class="math-container">$w(n)$</span> is its current weight and <span class="math-container">$x(i)$</span> is the <span class="math-container">$i$</span>th element of a training dataset, what should <span class="math-container">$\rho$</span> be?</p> <p>From what I have found online, <span class="math-container">$ρ$</span> is supposed to be <span class="math-container">$0 &lt; \rho &lt; \frac{2}{trace(X^TX)}$</span>, where <span class="math-container">$X$</span> is a matrix with all the training data the algorithm has processed at that point. One idea that I had, was to take <span class="math-container">$\rho = \frac{1}{trace(X^TX)} &lt; \frac{2}{trace(X^TX)}$</span>, but I do not know if that is correct. Also, one characteristic that this value has is that it changes with each iteration of the algorithm, as more samples are added to matrix <span class="math-container">$X$</span>.</p> <p>So, what is a good value for <span class="math-container">$\rho$</span>? Should it change during the execution of the algorithm or should it stay the same?</p>