Apologies for the lengthy title. My question is about the weight update rule for logistic regression using stochastic gradient descent.

I have just started experimenting on Logistic Regression. I came across two weight update expressions and did not know which one is more accurate and why they are different.

The first Method:

Source: (Book) Artificial Intelligence: A Modern Approach by Norvig, Russell on page 726-727: using the L2 loss function:

enter image description here

enter image description here

where g stands for the logistic function g' stands for g's derivative w stands for weight hw(x) represents the logistic regression hypothesis

The other method:

Source (Paper authored by Charles Elkan): Logistic Regression and Stochastic Gradient Training.

can be found here

enter image description here

  • $\begingroup$ How do you like that AI book? It’s on my reading list but I’ve done a fair amount of machine learning, deep learning, and now some RL, so I didn’t know how much would be new info. Would be interested in your take on it. $\endgroup$
    – Hanzy
    May 7 '19 at 18:58
  • $\begingroup$ @Hanzy Hi, well to be honest, my strategy is to search the subject at hand over several books. Thus, I don’t go through a whole book. However, what I likes about it is the clarity and it was used by an AI veteran of machinelearningmastery.com. $\endgroup$ May 8 '19 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.