I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general loopholes or reasons why this kind of behavior is exhibited?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ Hi and welcome to AI SE! Maybe it would be a good idea to provide an example of an algorithm that converges to one action, although I don't think a person needs this information to provide an answer (in this case). $\endgroup$– nbroCommented May 10, 2020 at 15:48
-
$\begingroup$ Hi @nbro , I have seen this happening in DQN or in policy gradient methods like PPO , so i believe there should be some generalized do's and dont's which we need to follow irrespective of algorithms to ensure that this issue doesnt happen $\endgroup$– JAYDEEP GHOSECommented May 10, 2020 at 15:51
-
$\begingroup$ Isn't the answer to your question just "Because there's probably just an optimal action for each state"? Why do you think this is an issue? It may not be an issue. It depends. If the environment is stochastic, that may be an issue, but if the environment is deterministic, it may be the case that one action is typically the best one. $\endgroup$– nbroCommented May 10, 2020 at 15:54
-
$\begingroup$ Well .. not all environments are deterministic . For example If my agent is playing pacman and it does only one action then it might just hit into a wall and do nothing to recover . I dont think thats an optimal action and I have seen this happening as well... Correct me If I am missing anything here $\endgroup$– JAYDEEP GHOSECommented May 10, 2020 at 15:59
-
$\begingroup$ Yes, you're right. Sometimes the policy shouldn't always select the same action. For example, if you're playing rock paper scissors, the optimal policy shouldn't always choose the same action, because that would make you lose against any intelligent player that recognises the pattern of the policy. However, in the cases you found, maybe there's primarily one optimal action for each state. I don't know because I don't know to which problems those algorithms that you found have been applied. $\endgroup$– nbroCommented May 10, 2020 at 16:17
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
2
Why do RL implementations converge on one action?
If the optimal policy shouldn't always select the same action in the same state, i.e., if the optimal policy isn't deterministic (e.g., in the case of the rock paper scissors, the optimal policy cannot be deterministic because any intelligent player would easily memorize your deterministic policy, so, after a while, you would always lose again that player), then there are a few things that you can do to make your policy more stochastic
Change the reward function. If your agent ends up selecting always the same action and you don't want that, it's probably because you're not giving it the right reinforcement signal (given that the agent selects the action that apparently will give it the highest reward in the long run).
Try to explore more during training. So, if you're using a behavior policy like $\epsilon$-greedy, you may want to increase your $\epsilon$ (i.e. probability of selecting a random action).
If you estimated the state-action value function (e.g. with Q-learning), maybe you derived the policy from it by selecting the best action, but, of course, that will make your policy deterministic. You may want to use e.g. softmax to derive the policy from the state-action value function (i.e. the probability of selecting an action is proportional to its value), although Q-learning assumes that your target policy is greedy with respect to the state-action value function.
If the optimal policy is supposed to be deterministic, then, if you find the optimal policy (which isn't probably the case), you will end up with an agent that always selects the same action. In that case, obviously, it's not a problem that the RL agent selects always the same optimal action.
-
$\begingroup$ Thanks ... I will have to explore how the reward can be manipulated in environments like OPen AI Gym .... For your 2nd point for policy gradients , normally e-greedy is not followed so i guess this will be more applicable for Q learning in general .. $\endgroup$ Commented May 10, 2020 at 17:58
-
$\begingroup$ the problem still remains the same - after all the modifications mentioned by you $\endgroup$ Commented May 23, 2020 at 8:06