2
$\begingroup$

My doubt is like this :


Suppose we have an MLP. In an MLP, as per the backprop algorithm (back-propagation algorithm), the correction applied to each weight is :

$$ w_{ij} := -\eta\frac{\partial E}{\partial w_{ij}}$$ ($\eta$ = learning rate, $E$ = error in the output, $w_{ij}$ = $i^{\text{th}}$ neuron in the $j^{\text{th}}$ row or layer)

Now, if we put an extra factor in the correction as:

$$ w_{ij} := -k\eta \frac{\partial E}{\partial w_{ij}}$$ ($k$ denotes the number of iterations at the time of correction)

how much will that factor affect the learning of the network ? Will it affect the convergence of the network such that it takes time to fit to the data ?

NB : I am only asking this as a doubt. I haven't tried any ML projects recently, so this is not related to anything I am doing.

$\endgroup$
2
$\begingroup$

If anything, you want the learning rate to decrease as the number of iterations increases.

When you're looking for a good spot and you're clueless, take large steps. When you've found a pretty good spot, take small steps, so you don't end up far away.

In other fields of machine learning, there are studies of how the learning rate should scale. For example, in traditional reinforcement learning methods, if $\alpha_i$ is the learning rate at step $i$, then we want to have the following two criteria, to make sure we get convergence to the optimal policy:

  1. $\sum_{i=0}^{\infty} \alpha_i = \infty$. This makes sure that, no matter how bad our initial experience was, we can eventually forget it and replace it with better information.
  2. $\sum_{i=0}^{\infty} \alpha_i^2 < \infty$. This guarantees eventual convergence.

A typical choice here is $\alpha_i = \frac{1}{1+i}$, which fits both criteria.

I am unaware of similar criteria for MLPs, but if you're going to modify the step sizes, I would follow a similar approach. Make the step sizes decrease, but not too fast.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I didn't aim anything - I just wanted to know what happens when an additional factor is introduced, @RobbyGoetschalckx $\endgroup$ – Spectre Sep 20 at 3:22
  • $\begingroup$ I tried to view it from the mathematical side . $\endgroup$ – Spectre Sep 20 at 3:23
  • $\begingroup$ Also, I only wanted to know what happens when the factor introduced is the number of iterations at the time of correction. $\endgroup$ – Spectre Sep 20 at 3:25
  • 1
    $\begingroup$ Yes, and from the mathematical side, you want to decrease the step size as you're going along. When you've already learned a lot, you want step size to be small, so you don't overshoot the mark. And when you're far away, you want step sizes to be large. $\endgroup$ – Robby Goetschalckx Sep 20 at 3:25
  • 1
    $\begingroup$ I answered exactly what happens if you do that. If you scale it by the number of iterations, you will get a very unstable process. $\endgroup$ – Robby Goetschalckx Sep 20 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.