My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.
Can anyone suggest what is going wrong?
I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?
The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!
Some relevant images are given at the end.
The q-learning part of code is given below:
env.reset()
while not done:
current_state = current_state_to_string(assign_obs_to_bins(obs, bins))
if np.random.uniform() < EPSILON:
act = env.action_space.sample()
best_q_value = return_max_from_dict(q[current_state], action = act)
else:
act, best_q_value = return_max_from_dict(q[current_state])
obs, reward, done, _ = env.step(act)
q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
cnt+=1
total_reward += reward