Apologies for the lengthy title. My question is about the weight update rule for logistic regression using stochastic gradient descent.

I have just started experimenting on Logistic Regression. I came across two weight update expressions and did not know which one is more accurate and why they are different.

The first Method:

Source: (Book) Artificial Intelligence: A Modern Approach by Norvig, Russell on page 726-727: using the L2 loss function:

enter image description here

enter image description here

where g stands for the logistic function g' stands for g's derivative w stands for weight hw(x) represents the logistic regression hypothesis

The other method:

Source (Paper authored by Charles Elkan): Logistic Regression and Stochastic Gradient Training.

can be found here

enter image description here

  • $\begingroup$ How do you like that AI book? It’s on my reading list but I’ve done a fair amount of machine learning, deep learning, and now some RL, so I didn’t know how much would be new info. Would be interested in your take on it. $\endgroup$
    – Hanzy
    Commented May 7, 2019 at 18:58
  • $\begingroup$ @Hanzy Hi, well to be honest, my strategy is to search the subject at hand over several books. Thus, I don’t go through a whole book. However, what I likes about it is the clarity and it was used by an AI veteran of machinelearningmastery.com. $\endgroup$
    Commented May 8, 2019 at 7:17


You must log in to answer this question.