2
$\begingroup$

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$

I am using an epsilon greedy policy. In this environment, we get a reward only if we succeed. So I explored with 100% until I have 50 successes. Then I saved the data of failures and success in different bins. Then I sampled (with replacement) from these bins and used them to train the Q network. However, no matter how long I train the agent doesn't seem to learn.

The code is available in Colab. I am doing this for a couple of days.

PS: I modified the code for SARSA and Expected SARSA; nothing works.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ In general it is quite hard to do code reviews and bug hunts, so you may not get any response. To improve your chances you should give more context on the site, so your question does not rely on people going off site and getting involved in your project. For instance, what diagnostics have you collected, and what are the results? Have you observed more than "the agent doesn't seem to learn" that is worth sharing? What is your state representation for input to the NN, and what its the rough NN architecture - layer sizes, output activation function, loss function etc $\endgroup$
    – Neil Slater
    Commented Feb 20, 2020 at 21:51
  • $\begingroup$ Frozen lake is somewhat harder to solve than expected since it's a very stochastic environment. You will have a hard time solving it with DQN. Try solving it with regular Q learning. Epsilon-greedy isn't ideal because you will get stuck in local optima and you won't get out since exploration will diminish. Try using UCB strategy which is generally better than e-greedy for tabular methods. Also, resetting counters completely after certain amount of episodes in UCB helps in this case. $\endgroup$
    – Brale
    Commented Feb 20, 2020 at 22:11
  • $\begingroup$ @Brale_ I tried normal Q learning. It did not train. In any case i will also try UCB. $\endgroup$
    – kosa
    Commented Feb 20, 2020 at 22:32
  • $\begingroup$ @NeilSlater I used NN with 2-hidden layers (size = 50)+'relu' activations which outputs 4 values(one for each action). I used Adam with a constant learning rate for minimizing MSE error. I did not use any features to represent the state. I just used the state as the input to the NN. $\endgroup$
    – kosa
    Commented Feb 20, 2020 at 22:40
  • 1
    $\begingroup$ @kosa If your regular Q learning algorithm does not work, then your DQN has no chance to work either. The neural network in DQN is just a storage device to replace the table for the Q-factors if you run out of memory. My hunch is that your target in test_fun is the real culprit. Currently it's just $Q^{new}$, it should be $(1-\alpha)Q^{old}+\alpha Q^{new}$. Also, frozen lake is a very simple problem (compared to Starcraft or Go), you won't need 2 hidden layers. $\endgroup$
    – Hai Nguyen
    Commented Feb 21, 2020 at 11:52

1 Answer 1

Reset to default
0
$\begingroup$

I see at least 3 issues with your DQN code that need to be fixed:

  1. You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.

  2. Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.

  3. Your batch_target is incorrect. You need to train on returns and not just rewards. In your train function, compute the 1-step return $r + \gamma \cdot max_{a'} Q(s',a')$, remembering to set $max_{a'} Q(s',a') = 0$ if $s'$ is terminal, and then pass it to model.fit() as your prediction target.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ The biggest remaining issue is probably this: "I just used the state as the input to the NN". Most neural networks are going to have poor performance handling unless the state values are in a nice range. $\endgroup$
    – Neil Slater
    Commented Feb 21, 2020 at 11:07
  • $\begingroup$ @NeilSlater I think is an issue. I will normalize the states. $\endgroup$
    – kosa
    Commented Feb 21, 2020 at 19:05
  • $\begingroup$ @Brett I separated them because of two reasons: (1) We get a reward = 1.0 only if we are successful. Until we have enough data on successes, the updates on Q(s, a) are redundant. (2) Moreover, the distribution of successes vs failures is uneven. However, in the end I just sampled from these bins with replacement, to a new bin, and trained on the Q function using the data in the new bin. PS: I have tried your way, it was not working at all. So, I came up with this implementation. $\endgroup$
    – kosa
    Commented Feb 21, 2020 at 19:12
  • $\begingroup$ @Brett I think I am following the same update rule. Please see the "test_fun" method in the Q_learn class. $\endgroup$
    – kosa
    Commented Feb 21, 2020 at 19:19
  • 1
    $\begingroup$ @kosa Even though those updates are redundant, they still need to be put into the replay memory because they change the probability of those samples being chosen. It's OK if they're uneven because they reflect the true probabilities of encountering those (s,a,r,s') transitions in the environment. This makes sure your Q-function is an unbiased estimate of the true returns. $\endgroup$
    – Brett Daley
    Commented Feb 23, 2020 at 21:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .