I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the following: +1 for covering a blank cell, and -1 per step. So, if the cell was colored after a step, the summed reward is (+1) + (-1) = 0, otherwise it is (0) + (-1) = -1. The environment is a tensor whose layers encode the positions to be covered and the position of the agent.

Under this reward scheme, DQN fails to find a solution (implementation: stable_baselines3). However, when the rewards are reduced by a factor of 10 to +0.1/-0.1, then the algorithm learns an optimal path.

I wonder why that happens. I have tried reducing learning rate and gradient clipping (by norm) for the first case to see whether it will improve the learning, but it does not.

The activation function used is ReLU

  • $\begingroup$ Likely to be a bug or problem with your implementation. First thought on what might be relevant: How is your DQN network structured, and what transform function is being applied to the output (is it linear, ReLU, sigmoid, tanh for example)? Please add that detail using edit. Also consider breaking up into shorter paragraphs, maybe show a picture of the environment etc. $\endgroup$ – Neil Slater Aug 24 '20 at 12:49
  • $\begingroup$ Another relevant point might be your state representation. When the history of actions is important it is also important to capture that somehow in the state. However, as you say scaling the rewards allows correct learning, it seems less likely to be that issue to me. $\endgroup$ – Neil Slater Aug 24 '20 at 12:52
  • $\begingroup$ @NeilSlater, I just unit tested my environment ensure that the rewards are assigned as described: everything is ok. Extended the description of the problem as well. I doubt there is an error in the implementation because I am using stable_baselines3. Re state representation and history, it is important, yes, but not relevant for this issue... $\endgroup$ – d56 Aug 24 '20 at 18:00
  • $\begingroup$ Are you using ReLU in the output layer? How many steps are there, and what are the expected returns from random policy and optimal policy (when the rewards are simply +1/-1)? Are you using any regularisation? $\endgroup$ – Neil Slater Aug 24 '20 at 18:30
  • $\begingroup$ Well, the output layer does not have ReLUs because I am using DQN and it should predict the state-action value. I am using 1-step DQN in the environment that has 51 cells to cover on a 100 cell grid, the maximum number of moves (steps) is capped at 200. No regularization used. Trying to reprocude stable-baselines3 behavior with my own code atm $\endgroup$ – d56 Aug 26 '20 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.