Can we solve the environment with only the linear and angular position through Q-Learning?

Question

I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was a train with 90 000 episodes and the agent didn't get more than 20 reward.

The algorithm that i'm using is the tabular method Q-Learning and Epsilon-greedy for action selection.

This is the code, i implement:

import gymnasium as gym
import numpy as np
import math

# Axis dimensions
max_x = 2.4
min_x = -2.4
max_theta = 12
min_theta = -12
x_bins = 24
theta_bins = 24

x_axis_space = np.linspace(min_x, max_x, x_bins)
theta_axis_space = np.linspace(min_theta, max_theta, theta_bins)

# Env setup - First state
env = gym.make('CartPole-v1')
obs, info = env.reset() 

last_state = state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))

# import ipdb; ipdb.set_trace()
# print(state)

# Hyperparameters
GAMMA = 0.99        # Discount factor (0.95 Looking harder for Long-term reward)
ALPHA = 0.1         # Learning rate
EPSILON = 1         # 100% of exploration
N_EPISODES = 4000
MAX_REWARD = 500
total_reward = 0
episode_reward = []
EPSILON = 1.0 
DECAY_RATIO = 1-0.00001


class CartPoleQAgent():
    def __init__(self, n_bins_x, n_bins_theta, n_actions):       # x  theta L//R
        self.n_bins_x = n_bins_x
        self.n_bins_theta = n_bins_theta
        self.n_actions = n_actions
        self.q_table = np.zeros((n_bins_x+1, n_bins_theta+1, n_actions))


def exp_dec_epsilon_greedy(q_table, state, finish_training):
    global EPSILON
    
    if np.random.random() > EPSILON or finish_training == 1:    # Exploit
        # Select the greedy action max Q
        max_q = q_table[state[0]][state[1]].max()
        for i in range(2):
            if max_q == q_table[state[0]][state[1]][i]:
                return i
    else:                                                       # Explore
        # Select a random action
        return env.action_space.sample()


def update_q_value(q_table, last_state, action, reward, state):
    global GAMMA, ALPHA
    # import ipdb; ipdb.set_trace()
    action = int(action)
    last_x = last_state[0]  
    last_theta = last_state[1]
    x = state[0] 
    theta = state[1]

    return (q_table[last_x][last_theta][action] + ALPHA*(reward + GAMMA*q_table[x][theta][action] - q_table[last_x][last_theta][action]))


if __name__ == "__main__":
    
    agent = CartPoleQAgent(24, 24, 2)
    finish_training = 0
    i_episode = 0
    mean_reward = 0
    
    while mean_reward < 500:
        i_episode_reward = 0  
        
        while True:     # End of an episode
            action = exp_dec_epsilon_greedy(agent.q_table, state, finish_training)
            
            result = env.step(action)  
            obs, reward, done, info = result[:4]
            i_episode_reward = i_episode_reward + reward

            if done:    # If the episode has ended
                env.reset() # Always the cartpole end conditions are met, to reboot the env
                break
            
            state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))
            agent.q_table[last_state[0]][last_state[1]][action] = update_q_value(agent.q_table, last_state, action, reward, state)
            last_state = state
        
        episode_reward.append(i_episode_reward)
        EPSILON = EPSILON * DECAY_RATIO 

        mean_reward = np.mean(episode_reward[len(episode_reward)-100:])    
        print("Episode: " + str(i_episode) + " Episode Reward: " + str(i_episode_reward) + " eps: " + str(EPSILON) + " Mean Reward: " + str(mean_reward))
        i_episode = i_episode + 1 
    
env.close()

Answer 1

This isn't possible with a basic tabular approach, or with any agent that has no internal memory. The velocities are a necessary part of the state. Without them, the state is only partially observable, and the difference is really important. For example, imagine the positions are neutral - the pole and cart are in the middle - there's a huge difference between the pole being still or moving fast in either direction. It would change which action was best. This is not universally true for all environments, but a good rule of thumb is if your actions are to apply forces (and therefore impulse or acceleration), then you will need to observe the first-order effect of the actions, i.e. the velocities.

The Q table that is learned by the agent as you currently have it set up will contain the mean action values that cover the distributions of the pole moving in either direction, for the best policy that the agent can come up with when it is not sure which direction the pole is moving in. This may still be optimal in some sense for the partially-observed state, but it will fall short of "solving" the environment.

How to address it:

• Simplest fix - add the velocity back into the state.

• Interesting, but hard, fix - use a function approximator with in-built memory for the Q network, such as an LSTM recurrent neural network. Having a memory will allow the agent to figure out velocity and the rules for it due to history of how the pendulum has moved.

The memory version may also be less effective than adding to the state, since the agent needs to observe at least one extra time step before it can approximate the velocity and determine the best action. However, the reason I say it is "interesting" is because it is a setup where the agent really does learn how to solve the harder challenge you have set it.

In addition, you may find policy gradient methods do slightly better than tabular Q learning, at least in terms of average return. This is because a policy gradient method can learn to "hedge its bets" and behave randomly when it lacks enough state information to make a clear decision.

Answer 2

Thanks man, I already solved the problem with Tabular Q-Learning without Neural Networks. I had to just add the cart and pole velocities to the state and the agent learn to balance the pendulum.