In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,\hat{y}_N]$ is close the target $\mathbf{y} = [y_1,y_2,...,y_N]$, the mean squared-error (MSE) loss function is: $$ \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2 $$ The parameters $\theta$ are then adjusted using the gradient descent update rule: $$ \theta_{k+1} \leftarrow \theta_{k} - \alpha \cdot \nabla \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}_{\theta_k}) $$ Where $\alpha$ is the learning rate. I am aware that if $\alpha$ is too small, the parameters $\theta$ might never converge, or is too slow to converge, to the optimal set of parameters $\theta^*$, and if $\alpha$ is too large, the iterates of $\theta$ could oscillate and also never converge. My question has to do with the latter scenario, where $\alpha$ is too big, which leads to overshooting and oscillation.

A good way of choosing $\alpha$ is using backtracking line search. However, because the neural network has many parameters, it is not practical to perform line search, and $\alpha$ needs to be chosen using another way.

Is it possible to allow a larger value of the learning rate before overshooting and oscillation by "elongating" valleys in the MSE loss function $\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$? More precisely, by transforming $\mathbf{y}$ and $\hat{\mathbf{y}}$ in some way before computing the loss? For example, in practice, I have found the following modification to the MSE loss function to be very helpful in avoiding overshooting and oscillation, even with large learning rates: $$ \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (\log(y_i + \epsilon) - \log(\hat{y}_i + \epsilon))^2 $$ Where $\epsilon$ is a small value. However, I am not sure why this modification helps.

  • $\begingroup$ So... $\log^2\frac{y_i}{\hat{y}_i}$ ? $\endgroup$
    – Kostya
    Commented Apr 17, 2021 at 19:13
  • $\begingroup$ Sure. This doesn't really explain why it helps to avoid overshooting though. $\endgroup$
    – mhdadk
    Commented Apr 17, 2021 at 19:21
  • 1
    $\begingroup$ The modified loss function possibly has much flatter gradient than the MSE loss, so the individual weight updates might be smaller, espcially if you are using an optimiser that has some kind of momentum. $\endgroup$
    – Mike NZ
    Commented Apr 18, 2021 at 8:32


You must log in to answer this question.