I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the following: +1 for covering a blank cell, and -1 per step. So, if the cell was colored after a step, the summed reward is (+1) + (-1) = 0, otherwise it is (0) + (-1) = -1. The environment is a tensor whose layers encode the positions to be covered and the position of the agent.
Under this reward scheme, DQN fails to find a solution (implementation: stable_baselines3). However, when the rewards are reduced by a factor of 10 to +0.1/-0.1, then the algorithm learns an optimal path.
I wonder why that happens. I have tried reducing learning rate and gradient clipping (by norm) for the first case to see whether it will improve the learning, but it does not.
The activation function used is ReLU