I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from either of these actions in the current state is very close). The "true" Q-values I know from an analytical solution to the problem.
I have full control over the MDP, including the reward function, which in my case is sparse (0 until the terminal episode). The rewards are the same for identical transitions. However, the rewards vary for any given state and action taken therein. Moreover, the environment is only stochastic for a part of the actions in the action space, i.e. the action chosen by the agent influences the stochasticity of the rewards.
How can I still ensure that the algorithm gets these values (and their relative ranking) right?
Currently, what happens is that the loss function on the Q-estimator decreases rapidly in the beginning, but then starts evening out. The Q-values also first converge quickly, but then start fluctuating around.
I've tried increasing the batch size, which I feel has helped a bit. What did not really help, however, was decreasing the learning rate parameter in the loss function optimizer.
Which other steps might be helpful in this situation?
So, the algorithm usually does find only a slightly suboptimal solution to the MDP.