What are the differences between stability and convergence in reinforcement learning?

Question

The terms are mentioned in the paper: An Emphatic Approach to the Problem of off-Policy Temporal-Difference Learning (Sutton, Mahmood, White; 2016) and more, of course.

In this paper, they proposed the proof of "stability" but not convergence.

It seems that stability is guaranteed if the "key matrix" is shown to be positive definite. However, convergence requires more than that.

I don't understand the exact difference between the two.

Answer 1

Sometimes when training, particularly in reinforcement learning, the model can become unstable due to the amount of variance that exists in the training data that the agent generates by interacting with the environment. This is certainly a problem at the start of training as you can get huge outliers in the data because the agent is behaving randomly. You can find that just one update to the policy could potentially make it collapse because it moves the policy into some obscure region, e.g. so the agent always take a particular action. You can make training more stable by using larger batches and a smaller learning rate so it takes smaller steps at a time, but the downside to that is training is slower. So you need to test different hyperparameters to find a good trade-off between the two. You can also use an training architecture such as Proximal Policy Optimization (PPO) which clips the amount the policy can move in any given update to try and maintain some stability.

Convergence is a term used to describe when the model has found an optimal policy and isn't learning any further, usually demonstrated when the reward plateaus for a certain number of episodes. Of course, it may have settled on a local optima, and other global optima may exist; the data you present and the way you train your model may yield better results - again, all part of testing and experimentation.