2
$\begingroup$

I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?

$\endgroup$

1 Answer 1

2
$\begingroup$

You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$ and the output would be 6 Q-values for frame $F_k$.

$\endgroup$
2
  • $\begingroup$ I still don't get the idea, when I input 4 frames It outputs 4 by 6 values, so what are these values? $\endgroup$ Jun 5, 2020 at 17:58
  • 1
    $\begingroup$ It outputs 4 by 6 matrix because you defined the network architecture such that it outputs 4 by 6 matrix. Tensoflow, Pytorch or whatever you used cannot read your mind and know how many outputs do you want. You need to fix your network architecture $\endgroup$
    – Brale
    Jun 5, 2020 at 18:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .