I've pondered this for a while without developing an intuition for the math behind the cause of this.
So what causes a model to need a low learning rate?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.
hence result in a higher loss.
Since Gradient descent can only find local minimum so, the lower learning rate may result in bad performance. To do so, it is better to start with the random value of the hyperparameter can increase model training time but there are advanced methods such as adaptive gradient descent can manage the training time.
There are lots of optimizer for the same task but no optimizer is perfect. It depends on some factors
PS. It is always better to go with different rounds of gradient descent