My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.
Can anyone suggest what is going wrong?
I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?
The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!
Some relevant images are given at the end.
The q-learning part of code is given below:
env.reset() while not done: current_state = current_state_to_string(assign_obs_to_bins(obs, bins)) if np.random.uniform() < EPSILON: act = env.action_space.sample() best_q_value = return_max_from_dict(q[current_state], action = act) else: act, best_q_value = return_max_from_dict(q[current_state]) obs, reward, done, _ = env.step(act) q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act]) cnt+=1 total_reward += reward