Consider the following data with one input (x) and one output (y):
(x=1, y=2)
(x=2, y=1)
(x=3, y=2)
Apply linear regression on this data, using the hypothesis $h_Θ(x) = Θ_0 + Θ_1 x$, where $Θ_0$ and $Θ_1$ represent the parameters to be learned. Considering the initial values $Θ_0$= 1.0, and $Θ_1$ = 0.0, and learning rate 0.1, what will be the values of $Θ_0$ and $Θ_1$ after the first three iterations of Gradient Descent

From least squares method I took the derivative with respect to $Θ_0$ and $Θ_1$ and plugged in the initial values to get the slope/intercept and multiplied it by the learning rate 0.1 to get the step size.The step size was used to calculate the new $Θ_0$ and $Θ_1$ values.

I am getting $Θ_0$ as 1.7821 when following the above. Please let me know if the approach followed and the solution correct or there is a better way to solve

X = np.array([1,2,3])
Y = np.array([2,1,2])

params = np.array([1, 0])

def loss(y, yhat):
    return ((y - yhat)**2).mean()

def model(x):
    return params[0] + params[1]*x

def loss_grad(y, yhat, x):
    return np.array([(2*(yhat-y)).mean(), (2*(yhat-y)*x).mean()])

lr = .1
for _ in range(3):
    yhat = model(X)
    l = loss(Y, yhat)
    g = loss_grad(Y, yhat, X)
    params = params - lr*g
    print(f'thetas are now {params} with new loss of {loss(Y, yhat)}')


thetas are now [1.13333333 0.26666667] with new loss of 0.6666666666666666
thetas are now [1.13333333 0.23111111] with new loss of 0.2696296296296296
thetas are now [1.14755556 0.22874074] with new loss of 0.262887242798354

I double checked this with keras too, but in numpy i explicitly wrote the gradients, i reccomend double checking your gradient or arithmetic


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.