# Is there a theoretical way to determine the best learning rate for gradient descent if the function is a simple known polynomial?

I was playing around gradient descent topic. Wrote a function that calculates a gradient descent of a degree-2 polynomial. While trying out what is the best "step size multiplyer" (a.k.a. "learning rate") I found that value of approximatively 0.7068 works best. I cannot figure out why it's best though- I simply found it by trial and error. Is it possible determine such values without any trial and error? Is there any mathematical theory that digs deep into this topic? Or is it either manual trial end error always? Or do I have to write a gradient descent over the gradient descent function in some way (with the "learning_rate" as input and the "number of steps" as output?

Quick search yields topic of Lipschitz constant, but it does not seem to provide a result similar to that value. The code in question:

def gradient_descent(derivative_function,
position,
step_multiplier=0.7068,
min_step=0.0001,
max_steps_num=1000):
current_step_num = 0
while True:
slope = derivative_function(position)
step = step_multiplier * slope
new_position = position - (step * step_multiplier)
if abs(step) < min_step:
print(f'Best result (step {current_step_num}): {new_position}')
return new_position
if current_step_num > max_steps_num:
print(f'Best result (step cap {current_step_num}): {new_position}')
return new_position
current_step_num += 1
position = new_position

# With step_multiplier=0.7068 an optimal value is found in just 3 steps
derivative_function=lambda a: 2 * (a - 1),
position=-10000,
step_multiplier=0.7068
)  # prints: Best result (step 3): 0.9999999943355022

# With step_multiplier=0.1 an optimal value is found after 833 steps
derivative_function=lambda a: 2 * (a - 1),
position=-10000,
step_multiplier=0.1
)  # prints: Best result (step 833): 0.9995185063601436
$$$$


• Thank you! Found the bug, it was the double step_multiplier` :D. Also indeed, with 0.5a multiplier it's just one step with a perfect result, vs 9 steps and an approximate result, for example, with a 0.6a multiplier. Will read through the link, thanks again. Commented Feb 22 at 18:41