I'm currently studying reinforcement learning (RL) and would like to understand non-stationary environments better. So for stationary environments, the Q-values of all state-action pairs converge after a certain number of episodes or trials (theoretically, after an infinite number of runs). This is seen when you plot the maximum change in the Q-table against the number of episodes. For a toy example it looked like this:

Convergence of Q-table in stationary environments

I understand there is no convergence for non-stationary problems as the environment keeps changing. In the stationary case, we can stop training once we observe convergence. But for the non-stationary case, we don't. What will be the right indicator to stop training in the non-stationary case? I've read dozens of papers on this but none really seems to discuss exactly when to stop training.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.