I am learning Reinforcement Learning from the lectures from David Silver. I finished lecture 6 and went on to try SARSA with linear function approximator for MountainCar-v0 environment from OpenAI.
A brief explanation of the MountainCar-v0 environment. The state is denoted by two features, position, and velocity. There are three actions for each state, accelerate forwards, don't accelerate, accelerate backward. The goal of the agent is to learn how to climb a mountain. The engine of the car is not strong enough to power directly to the top. So speed has to be built up by oscillating in the cliff.
I have used a linear function approximator, written by myself. I am attaching my code here for reference :-
class LinearFunctionApproximator:
''' A function approximator must have the following methods:-
constructor with num_states and num_actions
get_q_value
get_action
fit '''
def __init__(self, num_states, num_actions):
self.weights = np.zeros((num_states, num_actions))
self.num_states = num_states
self.num_actions = num_actions
def get_q_value(self, state, action):
return np.dot( np.transpose(self.weights), np.asarray(state) )[action]
def get_action(self, state, eps):
return randint(0, self.num_actions-1) if uniform(0, 1) < eps else np.argmax( np.dot(np.transpose(self.weights), np.asarray(state)) )
def fit(self, transitions, eps, gamma, learning_rate):
''' Every transition in transitions should be of type (state, action, reward, next_state) '''
gradient = np.zeros_like(self.weights)
for (state, action, reward, next_state) in transitions:
next_action = self.get_action(next_state, eps)
g_target = reward + gamma * self.get_q_value(next_state, next_action)
g_predicted = self.get_q_value(state, action)
gradient[:, action] += learning_rate * (g_target - g_predicted) * np.asarray(state)
gradient /= len(transitions)
self.weights += gradient
I have tested the gradient descent, and it works as expected. After every epoch, the mean squared error between current estimate of Q and TD-target reduces as expected.
Here is my code for SARSA :-
def SARSA(env, function_approximator, num_episodes=1000, eps=0.1, gamma=0.95, learning_rate=0.1, logging=False):
for episode in range(num_episodes):
transitions = []
state = env.reset()
done = False
while not done:
action = function_approximator.get_action(state, eps)
next_state, reward, done, info = env.step(action)
transitions.append( (state, action, reward, next_state) )
state = next_state
for i in range(10):
function_approximator.fit(transitions[::-1], eps, gamma, learning_rate)
if logging:
print('Episode', episode, ':', end=' ')
run_episode(env, function_approximator, eps, render=False, logging=True)
Basically, for every episode, I fit the linear function approximator to the current TD-target. I have also tried running fit just once per episode, but that also does not yield any winning episode. Fitting 10 times ensures that I am actually making some progress towards the TD-target, and also not overfitting.
However, after running over 5000 episodes, I do not get a single episode where the reward is greater than -200. Eventually, the algorithm choses one action, and somehow the Q-value of other actions is always lesser than this action.
# Now, let's see how the trained model does
env = gym.make('MountainCar-v0')
num_states = 2
num_actions = env.action_space.n
function_approximator = LinearFunctionApproximator(num_states, num_actions)
num_episodes = 2000
eps = 0
SARSA(env, function_approximator, num_episodes=num_episodes, eps=eps, logging=True)
I want to be more clear about this. Say action 2 is the one which is the action which gets selected always after say 1000 episodes. Action 0 and action 1 have somehow, for all states, have their Q-values reduced to a level which is never reached by action 2. So for a particular state, action 0 and action 1 may have Q-values of -69 and -69.2. The Q-value of action 2 will never drop below -65, even after running the 5000 episodes.
