1
$\begingroup$

I am pretty new to RL and I am trying to code a simple RL task with pytorch.

The goal/task is the following: The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$.

If $t_1$ equals 450 or 475 then it gets a reward, else he does not get a reward.

I am training the agent with DQN algorithm on a NN ( with: 2 Linear layes: fist layer n_in=1 n_out 128 and second layer n_in=128 and n_out=5):

observation space($t_i$) is 700 --> $t_i \in [0,700[$
action space ($\Delta_t$) is 5 --> ($\Delta_t \in [-50,-25,0,25,50]$)

epsilon_start=0.9#e-greedy threshold start value
epsilon_end=0.01#e-greedy threshold end value
epsilon_decay=200#e-greedy threshold decay learning_rate=0.001# NN
optimizer learning rate batch_size=64#Q-learning batch size

Unfortunately it does not seem to converge to the values $t_i=$ 450 or 475. I doesn't seem to care about getting a reward. How can I improve my code so that the agent learns what I am trying to teach him? I put my code below in case the explanations were not clear enough:

from gym import spaces

class RL_env(gym.Env):
    metadata = {'render.modes': ['human']}

    
    def __init__(self):
        super(RL_env, self).__init__()
        
        n_actions_delta = 1 #delta_t
        self.action_space = spaces.Discrete(5)
        
        n_observations = 1 #time
    
        self.observation_space = spaces.Discrete(700)
       
        #initial time
        self.time = 0
        
        self.done = 0
        self.reward = 0

    def reset(self):
        self.reward = 0
        self.done = False
        return self.reward
       
    def step(self,delta_t):
        print('self time',self.time)
        d_t = np.arange(-50,70,25)
        
        self.time = (self.time + d_t[delta_t])%700
        print('delta time',d_t[delta_t],'-->','self time',self.time)
        
       
        
        if self.time == 475 or self.time == 450:
            self.reward = 1
            
            
        else:
            self.reward += 0
        
            
        info = {}
        print('total reward',self.reward)
        print('\n')
        return self.time,self.reward, self.done, info
    

    
    
    def render(self, mode='human', close=False):
        print()
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.autograd import Variable
from torch.distributions import Categorical
dtype = torch.float
device = torch.device("cpu")
import random
import math
import sys
if not sys.warnoptions:#igrnore warnings
    import warnings
    warnings.simplefilter("ignore")

#hyper parameters
epsilon_start=0.9
#e-greedy threshold start value
epsilon_end=0.01#e-greedy threshold end value
epsilon_decay=200#e-greedy threshold decay
learning_rate=0.001# NN optimizer learning rate
batch_size=64#Q-learning batch size 

env = RL_env()


#use replay memory (-> to stabilize and improve our algorithm)for training: store transitions observed by agent,
#then reuse this data later
#sample from it randomly (batch built by transitions are decorrelated)
class ReplayMemory:#allowes the agent to learn from earlier memories (speed up learning and break undesirable temporal correlations)
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
    def push(self, transition):#saves transition
        self.memory.append(transition)
        if len(self.memory)>self.capacity:#if length of memory arra is larger than capacity (fixed)
            del self.memory[0]#remove 0th element

    def sample(self, batch_number):#samples randomly a transition to build batch
        return random.sample(self.memory, batch_number)

    def __len__(self):
        return len(self.memory)
    
#Dqn NN (we want to maximize the discounted, cumulative reward)
#idea of Q-learning: we want to approximate with NN maximal Q-function (gives max return of action in given state)
#training update rule: use the fact that every Q-function for some policy obeys the Bellman equation
#difference between the two sides of the equality is known as the temporal difference error (we want to min -> Huber loss)
#calculate over batch of transitions sampled from the replay memory
class DqnNet(nn.Module):
    def __init__(self):
        super(DqnNet, self).__init__()
        
        state_space = 1
        action_space = env.action_space.n
        num_hid = 128
        self.fc1 = nn.Linear(state_space, num_hid)
        self.fc2 = nn.Linear(num_hid, action_space)
        self.gamma=0.5 #Q-learning discount factor (ensures that reward sum converges, 
                        #makes actions from far future less important)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        return x

#select action accordingly to epsilon greedy policy
#sometimes we use model for choosing action, other times sample uniformly 
#probability of choosing a random action will start at epsilon_start and will decay (epsilon_decay) exponentially
#towards epsilon_end
steps_done=0
def predict_action(state):
    global steps_done
    sample=random.random()#random number
    eps_threshold=epsilon_end+(epsilon_start-epsilon_end)*math.exp(-1.*steps_done/epsilon_decay)
    steps_done += 1
    if sample>eps_threshold:  
        x  = eps_threshold,model(Variable(state,).type(torch.FloatTensor)).data.max(0)[1].view(1, 1)
        return x#chose action from model
    
    else:
        x = eps_threshold,torch.tensor([[random.randrange(env.action_space.n)]])
        return x#choose random action uniformly

#wtih the update_policy function we perform a single step of the optimization
#first sample a batch, concatenate all the tensors into a single one, compute Q-value and max Q-value, 
#and combine them into loss
def update_policy():
    if len(memory)<batch_size:#we want to sample a batch of size 64
        return
    transitions = memory.sample(batch_size)#take random transition batch from experience replay memory
    batch_state, batch_action, batch_next_state, batch_reward = zip(*transitions)#convert batch-array of Transitions
                                                                              #to Transition of batch-arrays   
    #-->zip(*) takes iterables as arguments and return iterator
    
    batch_state = Variable(torch.cat(batch_state))#concatenate given sequence tensors in the given dimension
    batch_state = batch_state.resize(batch_size,1)
    batch_action = Variable(torch.cat(batch_action))
    batch_next_state = Variable(torch.cat(batch_next_state))
    batch_next_state = batch_next_state.reshape(batch_size,1)
    batch_reward = Variable(torch.cat(batch_reward))
    
    #print('model batch state',model(Variable(batch_state[0])))
    current_q_values = model(batch_state).gather(1, batch_action)#current Q-values estimated for all actions,
                                                                 #compute Q, then select the columns of actions taken,
                                                                 #these are the actions which would've been taken
                                                                 #for each batch state according to policy_net
    max_next_q_values = model(batch_next_state).detach().max(1)[0]#predicted Q-values for non-final-next-states
                                                                  #(-> gives max Q)
    expected_q_values = batch_reward + (model.gamma * max_next_q_values)

    #loss is measured from error between current and newly expected Q values (Huber Loss)
    loss = F.smooth_l1_loss(current_q_values, expected_q_values)

    # backpropagation of loss to NN --> optimize model
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss, np.sum(expected_q_values.numpy())
    

def train(episodes):
    scores = []
    Losses = []
    Bellman = []
    Epsilon = []
    Times = []
    Deltas = []
    
    
    
    for episode in range(episodes):  
        state=env.reset()#reset environment
        print('\n')
        print('episode',episode)
           
        epsilon_action = predict_action(torch.FloatTensor([state]))
        
        action = epsilon_action[1] #after each time step predict action

        next_state, reward, done,info = env.step(action.item())#step through environment using chosen action
        
        epsilon = epsilon_action[0]
        Epsilon.append(epsilon)
        print(reward,'reward')
              
        state=next_state
        Times.append(state)
        scores.append(reward)            

    
        memory.push((torch.FloatTensor([state]),action,torch.FloatTensor([next_state]),
                         torch.FloatTensor([reward])))#action is already a tensor
        up = update_policy()#update_policy()#update policy
            
        if up != None:
            Losses.append(Variable(up[0]))
            print('loss',Variable(up[0]))
            Bellman.append(up[1])

        #calculate score to determine when the environment has been solved
        mean_score=np.mean(scores[-50:])#mean of score of last 50 episodes
        #every 50th episode print score
        if episode%50 == 0:
            print('Episode {}\tScore: {}\tAverage score(last 50 episodes): {:.2f}'.format(episode,scores[-50:],mean_score))

    
    #print('Losses',Losses)
    Losses = torch.stack(Losses).numpy()
    #print('Losses',Losses)
    plt.plot(np.arange(len(Losses)),Losses)
    plt.xlabel('Training iterations')
    plt.ylabel('Loss')
    plt.show()
    
    Bellman = np.array(Bellman)
    #print('Bellman',Bellman,'\n')
    plt.plot(np.arange(len(Bellman)),Bellman)
    plt.xlabel('Training iterations')
    plt.ylabel('Bellman target')
    plt.show()
    
    #print('scores',scores)
    plt.plot(np.arange(len(scores)),scores)
    plt.xlabel('Training iterations')
    plt.ylabel('Reward')
    plt.show()
    
    #print('epsilon',Epsilon)
    plt.plot(np.arange(len(Epsilon)),Epsilon)
    plt.xlabel('Training iterations')
    plt.ylabel('Epsilon')
    plt.show()
    
    print('Times',Times[-25:])
    print('Deltas',Deltas[-25:])
    
    Times = np.array(Times)
    print('Times',Times)
    #plt.figure(figsize=(31,20))
    plt.figure(figsize=(9,7))
    plt.plot(np.arange(len(Times)),(np.array(Times)))
    plt.xlabel('Training iterations')
    plt.ylabel('t')
    plt.show()
     
    Times_1 = np.array(Times[-300:])
    print('t',Times)
    plt.figure(figsize=(9,7))
    plt.plot(np.arange(len(Times_1)),(np.array(Times_1)))
    plt.xlabel('Last 300 Training iterations')
    plt.ylabel('t')
    plt.ylim(0,1000)
    plt.show()
    
model = DqnNet()#policy         
memory = ReplayMemory(20000)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

train(10000)
$\endgroup$
1
$\begingroup$

I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits $t=475$ for example, the reward given by the environment increases by 1. So if the agent oscillates between $t=450$ and $t=475$, at each timestep the environment gives a greater and greater reward. Your Q values have no reason to converge. Instead of incrementing the reward, you should just return a reward of $1$ at those two desired states and return $0$ otherwise.

Secondly, it appears that you don't reset the state during reset(), I think you should have self.time = 0 in there (unless this was intentional).

Edit : I also question why you're using DQN for this problem. This problem should be very easy to solve with standard (tabular) Q Learning.

Edit 2 : Your training loop looks wrong. You reset the environment at each iteration in your loop, so your episodes are one timestep long each. What I think you want instead is to have another loop under reset() something like

for episode in range(episodes):
  s = env.reset()
  episode_score = 0 
  for step in range(max_timesteps):
    a = agent.predict_action(s)
    ns, r, d, _ = env.step(a)
    episode_score += r
   ... 
    agent.update_policy()
   ... 
  scores.append(episode_score)
 ... 
$\endgroup$
7
  • $\begingroup$ thanks, I just corrected the typo in the reset function and changed the reward scheme.. unfortunately, still no convergence towards t = 475 or 450 ... and to answer your question: i am still new to reinforcement learning, and wanted to learn how to implement DQN on a simple problem, just for the sake of learning and understanding $\endgroup$ – HELP Oct 11 '20 at 15:31
  • $\begingroup$ I also suggest you increase gamma. Gamma=0.5 is fairly low, and it's very possible that the rewards are just not propagating back to other states due to the exponential decay. Try with gamma=0.99 or something. Not sure if this will be the definitive solution, but it can't hurt. $\endgroup$ – harwiltz Oct 11 '20 at 17:44
  • $\begingroup$ Also, can you describe the results you get? As in, what do you mean by "no convergence towards t=..."? Can you talk about what you expect vs what you observe? $\endgroup$ – harwiltz Oct 11 '20 at 17:46
  • $\begingroup$ thank you, so let me explain: in the end i want the agent to do the following 475 +0 = 475 --> 475 + 0 = 475 --> 475 +0 = 475.... OR something like 475 -25 = 450 --> 450 +25 = 475 --> 475 -25 = 450 .... (i want the sum to always be 475 or 450)... what I observe: it just takes the same action for any state ( for example 450 +25 = 475 --> 475 + 25 = 500 --> 525 + 25 = 550 --> 550 + 25 = 575....).. so it just learns to take always the same action over and over again without caring about the result $\endgroup$ – HELP Oct 11 '20 at 19:35
  • $\begingroup$ Noticed another issue, updated my original answer. $\endgroup$ – harwiltz Oct 11 '20 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.