5
$\begingroup$

I'm new to neural network, I study electrical engineering, and I just started working with ADALINEs.

I use Matlab, and in their Documentation they cite :

However, here the LMS (least mean squares) learning rule, which is much more powerful than the perceptron learning rule, is used. The LMS, or Widrow-Hoff, learning rule minimizes the mean square error and thus moves the decision boundaries as far as it can from the training patterns.

The LMS algorithm is the default learning rule to linear neural network in Matlab, but a few days later I came across another algorithm which is : Recursive Least Squares (RLS) in a 2017 Research Article by Sachin Devassy and Bhim Singh in the journal: IET Renewable Power Generation, under the title : Performance analysis of proportional resonant and ADALINE-based solar photovoltaic-integrated unified active power filter where they state :

ADALINE-based approach is an efficient method for extracting fundamental component of load active current as no additional transformation and inverse transformations are required. The various adaptation algorithms include least mean square, recursive least squares etc.

My questions are:

  • Is RLS just like LMS (i mean can it be used as a learning algorithm too)?
  • If yes, how can I customize my ADALINE to use RLS instead of LMS as a learning algorithm (preferably in Matlab, if not in Python) because I want to do a comparative study between the two Algorithm!
Improve this question
$\endgroup$
4
  • $\begingroup$ Really you want use an Adlines NN instead of perceptrons ? Sorry for the a few off-topic comment, but surprises me. $\endgroup$
    – pasaba por aqui
    Commented Mar 17, 2018 at 9:39
  • $\begingroup$ LMS and RLS are not learning rules, but cost functions. RLS th same than LMS after apply a filter to the error measures, usually a forget factor. $\endgroup$
    – pasaba por aqui
    Commented Mar 17, 2018 at 9:42
  • $\begingroup$ Please feel free to comment, i'm new so honestly i don't know, if there is something you want to suggest i'm listening, thank you $\endgroup$
    – Carter Nolan
    Commented Mar 17, 2018 at 9:44
  • 1
    $\begingroup$ If there are not a non-linear element we are more in the area of linear filtering than in the one of AI. Moreover, as combination of linear elements can be flatten, multi-layer concept disappears. Non-linear activation functions as Heaviside (perceptron), sigmoid, ... are the center of the NN. $\endgroup$
    – pasaba por aqui
    Commented Mar 17, 2018 at 10:42

1 Answer 1

Reset to default
1
$\begingroup$

RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section "8.6 Approximate Second-Order Methods" of the following book available online:

https://www.deeplearningbook.org/contents/optimization.html

Deep Learning An MIT Press book Ian Goodfellow and Yoshua Bengio and Aaron Courville

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .