I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?
1 Answer
$\begingroup$
$\endgroup$
2
You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$ and the output would be 6 Q-values for frame $F_k$.
-
$\begingroup$ I still don't get the idea, when I input 4 frames It outputs 4 by 6 values, so what are these values? $\endgroup$ Jun 5, 2020 at 17:58
-
1$\begingroup$ It outputs 4 by 6 matrix because you defined the network architecture such that it outputs 4 by 6 matrix. Tensoflow, Pytorch or whatever you used cannot read your mind and know how many outputs do you want. You need to fix your network architecture $\endgroup$– BraleJun 5, 2020 at 18:07