0
$\begingroup$

I'm trying to solve the openai gym taxi problem (v3) using deep q learning. I've already had some success with the q-table approach, but for the life of me cannot manage to train a NN to learn a reasonable action policy. I'm doing the training using an AWS p3.2xlarge instance.

My approach is fairly straightforward, I set up the environment and agent, then run the training loop.

My code more or less looks like this:

import gym
from taxi_agent import Agent

env = gym.make('Taxi-v3').env
optimizer = Adam(learning_rate=0.001)
agent = Agent(env, optimizer)

batch_size = 32
num_of_episodes = 200
timesteps_per_episode = 120

The agent was cobbled together from various examples online:

import numpy as np
import random
from IPython.display import clear_output
from collections import deque
from tensorflow.keras import Model, Sequential
from tensorflow.keras.layers import Dense, Embedding, Reshape
from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard


class Agent:
    def __init__(self, environment, optimizer):
        
        # Initialize atributes
        self._state_size = environment.observation_space.n
        self._action_size = environment.action_space.n
        self._optimizer = optimizer
        
        self.expirience_replay = deque(maxlen=2000)
        
        # Initialize discount and exploration rate
        self.gamma = 0.6
        self.epsilon = 0.5
        
        # Build networks
        self.q_network = self._build_compile_model()
        self.target_network = self._build_compile_model()
        self.align_target_model()

        #: Set up some callbacks
        self.checkpoint_filepath = 'checkpoints/'
        model_checkpoint_callback = ModelCheckpoint(
            filepath=self.checkpoint_filepath,
            save_weights_only=True,
            save_freq='epoch'
        )
        # tensorboard_callback = TensorBoard('logs', update_freq=1)
        self.model_callbacks = [model_checkpoint_callback]
        
        self.history = []

    def store(self, state, action, reward, next_state, terminated):
        self.expirience_replay.append((state, action, reward, next_state, terminated))
    
    def _build_compile_model(self):
        model = Sequential()
        model.add(Embedding(self._state_size, 10, input_length=1))
        model.add(Reshape((10,)))
        model.add(Dense(48, activation='tanh'))
        model.add(Dense(24, activation='tanh'))
        model.add(Dense(self._action_size, activation='linear'))
        
        model.compile(loss='mse', optimizer=self._optimizer)
        return model

    def restore_weights(self):
        path = self.checkpoint_filepath
        print(f"restoring model weights from {path}")
        self.q_network.load_weights(path)

    def align_target_model(self):
        self.target_network.set_weights(self.q_network.get_weights())
    
    def act(self, state, environment):
        if np.random.rand() <= self.epsilon:
            return environment.action_space.sample()
        
        q_values = self.q_network.predict(state)
        return np.argmax(q_values[0])

    def retrain(self, batch_size, epochs=1):
        minibatch = random.sample(self.expirience_replay, batch_size)
        
        for state, action, reward, next_state, terminated in minibatch:
            
            target = self.q_network.predict(state)
            
            if terminated:
                target[0][action] = reward
            else:
                t = self.target_network.predict(next_state)
                target[0][action] = reward + self.gamma * np.amax(t)
            
            history = self.q_network.fit(state, target, epochs=1, verbose=0, callbacks=self.model_callbacks)
            self.history.append(history.history)

The training loop uses the agent to act in the environment up to a number of batch_size actions. Next, it retrains the model based on a random sample of the experience for every subsequent timestep.

I have it set to print out feedback whenever the environment terminates (achieves the objective). In practice this never happens.

I've reloaded trained models from weights and trained for cumulative 24 hours without much success. I've also tried silly things like updating the target network after N steps just so it learns something - no luck.

If I try to use my trained model to solve an example env instance, it just wants to move south other than the random actions it set to do 50% of the time.

It would be great it someone could give me some advice towards what to try next. I can keep playing around with hyperparameters but I don't have the best intuition around where to optimize my efforts.

iterations = 0
state = env.reset()
env.render()
while not terminated:
    state = np.reshape(state, [1, 1])
    action = agent.act(next_state, env)
    next_state, reward, terminated, info = env.step(action) 
    next_state = state
    if iterations % 10: env.render()
    iterations += 1
    if iterations > 1000: break
$\endgroup$
2
  • 1
    $\begingroup$ Could you please put your specific question in the title so that people immediately understand what your specific question is? $\endgroup$ – nbro Apr 15 at 10:23
  • $\begingroup$ I added more detail to the title. $\endgroup$ – Sledge Apr 15 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.