1
$\begingroup$

My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.

Can anyone suggest what is going wrong?

I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?

The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!

Some relevant images are given at the end.

The q-learning part of code is given below:

env.reset()
while not done:    
    current_state = current_state_to_string(assign_obs_to_bins(obs, bins))

    if np.random.uniform() < EPSILON:
        act = env.action_space.sample()
        best_q_value = return_max_from_dict(q[current_state], action = act)
    else:
        act, best_q_value = return_max_from_dict(q[current_state])

    obs, reward, done, _  = env.step(act)
    q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
    cnt+=1
    total_reward += reward

States' Histogram

Reward after all episodes

Improve this question
$\endgroup$
5
  • $\begingroup$ I would like to add one more thing that I tried epsilon-greed approach, greedy approach, and random approach. All gave me the same result in terms of reward and in all cases, most of the states remained untouched. $\endgroup$ – Kashan Jul 27 '20 at 22:24
  • $\begingroup$ Why are you using Q learning rather than DQN? $\endgroup$ – David Ireland Jul 27 '20 at 23:49
  • $\begingroup$ because I am a beginner and do not want to ride a sports bike before properly learning to balance on the bicycle :) $\endgroup$ – Kashan Jul 28 '20 at 5:48
  • $\begingroup$ try to test your implementation on a simple grid world and see if it works. After that you need to properly discretize state space. Try using simple case first, use only 2 states, one state when pole is to the left of the 90 degrees, and second when its right to 90 degrees. Then keep refining your discretization if your performance is not good enough. $\endgroup$ – Brale Jul 28 '20 at 6:45
  • $\begingroup$ CartPole will not be solvable with tabular DQN without some tuning of the state space as @brale mentioned. Just do some of the exercises from the Sutton and Barto book if you want to practice Tabular Q-Learning. $\endgroup$ – David Ireland Jul 28 '20 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.