2
$\begingroup$

I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works.

Let's say we are playing Atari Breakout where we have only 3 actions: left, stay still, right. We have 2 networks- the policy_net and the target_net (technically both are the same) and they give 3 outputs which are the q values for the 3 actions. During exploration we choose:

random.randrange(3)

and during exploitation, we choose:

argmax(policy_net output)

where the input of the policy net is the current state.

Now, during each timeline of each episode, we are storing current_state, action, reward, and next_state in storage that we will later randomly shuffle and use in training, which we call experience replay memory. During training, let's say we extract a batch of (current_states, actions, rewards, next_states). Now, we get current_q_val and next_q_val as:

current_q_values = policy_net(current_states)
next_q_values = target_net(next_states)

We use the following equation to find the loss:

$$q_*(s,a) - q(s,a) = \text{loss}$$

$$E[R_{t+1} + \gamma \max_{a'} q_*(s', a')] - E[\Sigma_{k=0}^ \inf \gamma^k R_{t+k+1}] = \text{loss}$$

And for that, we find which one from the next_q_val is the best one and that we call $\max q_*(s',a')$ (we already have $q(s,a)=\text{current_q_values}$).

Now my question is, which is $R_{t+1}$ here? As we are taking a random batch from the experience replay memory, we don't have any specific time here and we cannot calculate the $R_{t+1}$ from the time $t$. And if we simply use $q*(s,a) - q(s,a)$ or $\text{next_q_val} - \text{current_q_val}$, then where is the importance of the reward? I don't really understand how we are using the rewards in the training. I mean, where are we making sure the positive and negative influences of good and bad rewards respectively? The fact that the agent takes an action (randomly or from the policy_net) which then gives a reward, I don't understand how to use this reward in the loss function so as to influence how the agent should take action given a state.

$\endgroup$
6
  • $\begingroup$ Where did you get the loss function from? The equation you give is a kind of conceptual description of "regret" in RL, and not used as a loss function in DQN in any meaningful way. $\endgroup$ Nov 18 '21 at 8:46
  • $\begingroup$ Hi @NeilSlater, I got the loss function from one of the youtube series on RL I watched. can you tell me what should be the corrected loss function? In the video series, they are using MSE loss on current_q_value and next_q_value $\endgroup$ Nov 18 '21 at 10:35
  • $\begingroup$ Hello. As I asked you on Stack Overflow, please, edit your post to format the math symbols with mathjax. You need to wrap them with $ on either side, like this $\pi$ to produce $\pi$. $\endgroup$
    – nbro
    Nov 18 '21 at 11:02
  • $\begingroup$ "From one of the youtube series" is too vague for me to correct it. Either the video is wrong or your understanding of it is wrong. Using MSE loss for the neural network versus the TD targets calculated from each minibatch drives the learning, but it is important to note that is a moving target. It is possible for the MSE loss to get larger, but the agent behaviour to improve at the same time, at least in the short term. That is because the MSE loss does not capture the full RL objective, just the NN's role's objective. $\endgroup$ Nov 18 '21 at 11:13
  • $\begingroup$ @nbro, okay I changed the equation $\endgroup$ Nov 18 '21 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.