1
$\begingroup$

How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch, and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)

But is this the same in DQNs? So, you sample a batch from your memory, say 64 transitions. Do I iterate through each transition and adjust the weights "on the fly", or do I calculate the average loss of the batch and THEN in a big step adjust the weights w.r.t the average batch loss?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t the average loss? (Please correct me if I'm wrong)

You are wrong.

The weights are adjusted w.r.t. the average gradient, and this must be calculated using individual loss function results. The average loss (or cost function when considering the whole dataset) is a useful metric for current performance, and it is the measure being minimised. But you cannot calculate meaningful gradients against the average loss directly.

But is this the same in DQNs?

The batch process is not as you described, but an experience replay minibatch in RL and a sampled minibatch in supervised learning can be very similar. The main difference in RL is that your prediction targets must be recalculated as part of the sampling process (using $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}\hat{q}(S_{t+1},a', \theta)$ to calculate the TD target, assuming you are using single step Q learning), whilst in most supervised learning the target values are fixed for each example.

In theory you could use repeated single item stochastic gradient descent in DQN, it doesn't break any theory, and it would work. However, it will usually be more efficient to use a standard minibatch update, combining all gradients into one average gradient for the minibatch and making a single update step.

If you are using a high level library for your neural network model in DQN, you usually don't need to worry about this detail. You can use the .fit function or whatever the library provides. In that case the only difference between a supervised learning update and an experience replay DQN update is what you get from the sampling. In supervised learning you get a set of $(\mathbf{x}_i, \mathbf{y}_i)$ examples directly by sampling a minibatch. In RL you get $(\mathbf{s}_i, \mathbf{a}_i, r, \mathbf{s'}_i, done)$ and must construct the $(\mathbf{x}_i, \mathbf{y}_i)$ minibatch from these before passing to your .fit function

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for clarifying. What exactly do you mean with "batch update"? $\endgroup$
    – Voß
    Commented Jan 15, 2020 at 10:32
  • $\begingroup$ @bruh: I meant minibatch update, as in the process you areasking about. Will edit $\endgroup$
    – Neil Slater
    Commented Jan 15, 2020 at 10:39
  • $\begingroup$ To get the average gradient do I calculate for every example the gradient for every wheight, then average it over the mini-batch and then adjust the weights with the averaged gradient? $\endgroup$
    – Voß
    Commented Jan 15, 2020 at 10:46
  • $\begingroup$ @bruh: Yes, that is how you would do it. Most libraries with a .fit method or similar will do that for you $\endgroup$
    – Neil Slater
    Commented Jan 15, 2020 at 10:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .