In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,\hat{y}_N]$ is close the target $\mathbf{y} = [y_1,y_2,...,y_N]$, the mean squared-error (MSE) loss function is: $$ \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2 $$ The parameters $\theta$ are then adjusted using the gradient descent update rule: $$ \theta_{k+1} \leftarrow \theta_{k} - \alpha \cdot \nabla \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}_{\theta_k}) $$ Where $\alpha$ is the learning rate. I am aware that if $\alpha$ is too small, the parameters $\theta$ might never converge, or is too slow to converge, to the optimal set of parameters $\theta^*$, and if $\alpha$ is too large, the iterates of $\theta$ could oscillate and also never converge. My question has to do with the latter scenario, where $\alpha$ is too big, which leads to overshooting and oscillation.

A good way of choosing $\alpha$ is using backtracking line search. However, because the neural network has many parameters, it is not practical to perform line search, and $\alpha$ needs to be chosen using another way.

Is it possible to allow a larger value of the learning rate before overshooting and oscillation by "elongating" valleys in the MSE loss function $\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$? More precisely, by transforming $\mathbf{y}$ and $\hat{\mathbf{y}}$ in some way before computing the loss? For example, in practice, I have found the following modification to the MSE loss function to be very helpful in avoiding overshooting and oscillation, even with large learning rates: $$ \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (\log(y_i + \epsilon) - \log(\hat{y}_i + \epsilon))^2 $$ Where $\epsilon$ is a small value. However, I am not sure why this modification helps.

  • $\begingroup$ So... $\log^2\frac{y_i}{\hat{y}_i}$ ? $\endgroup$ – Kostya Apr 17 at 19:13
  • $\begingroup$ Sure. This doesn't really explain why it helps to avoid overshooting though. $\endgroup$ – mhdadk Apr 17 at 19:21
  • 1
    $\begingroup$ The modified loss function possibly has much flatter gradient than the MSE loss, so the individual weight updates might be smaller, espcially if you are using an optimiser that has some kind of momentum. $\endgroup$ – Mike NZ Apr 18 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.