1
$\begingroup$

My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.

Can anyone suggest what is going wrong?

I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?

The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!

Some relevant images are given at the end.

The q-learning part of code is given below:

env.reset()
while not done:    
    current_state = current_state_to_string(assign_obs_to_bins(obs, bins))

    if np.random.uniform() < EPSILON:
        act = env.action_space.sample()
        best_q_value = return_max_from_dict(q[current_state], action = act)
    else:
        act, best_q_value = return_max_from_dict(q[current_state])

    obs, reward, done, _  = env.step(act)
    q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
    cnt+=1
    total_reward += reward

States' Histogram

Reward after all episodes

Improve this question
$\endgroup$
5
  • $\begingroup$ I would like to add one more thing that I tried epsilon-greed approach, greedy approach, and random approach. All gave me the same result in terms of reward and in all cases, most of the states remained untouched. $\endgroup$
    – SJa
    Jul 27, 2020 at 22:24
  • $\begingroup$ Why are you using Q learning rather than DQN? $\endgroup$
    – David Ireland
    Jul 27, 2020 at 23:49
  • $\begingroup$ because I am a beginner and do not want to ride a sports bike before properly learning to balance on the bicycle :) $\endgroup$
    – SJa
    Jul 28, 2020 at 5:48
  • $\begingroup$ try to test your implementation on a simple grid world and see if it works. After that you need to properly discretize state space. Try using simple case first, use only 2 states, one state when pole is to the left of the 90 degrees, and second when its right to 90 degrees. Then keep refining your discretization if your performance is not good enough. $\endgroup$
    – Brale
    Jul 28, 2020 at 6:45
  • $\begingroup$ CartPole will not be solvable with tabular DQN without some tuning of the state space as @brale mentioned. Just do some of the exercises from the Sutton and Barto book if you want to practice Tabular Q-Learning. $\endgroup$
    – David Ireland
    Jul 28, 2020 at 8:49

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .