I'm currently studying reinforcement learning (RL) and would like to understand non-stationary environments better. So for stationary environments, the Q-values of all state-action pairs converge after a certain number of episodes or trials (theoretically, after an infinite number of runs). This is seen when you plot the maximum change in the Q-table against the number of episodes. For a toy example it looked like this:
I understand there is no convergence for non-stationary problems as the environment keeps changing. In the stationary case, we can stop training once we observe convergence. But for the non-stationary case, we don't. What will be the right indicator to stop training in the non-stationary case? I've read dozens of papers on this but none really seems to discuss exactly when to stop training.