I know that deepmind used deep Q learning (DQN) for its Atari game AI. It used a conv neural network (CNN) to approximate Q(s,a)
from pixels instead of from a Q-table. I want to know how DQN converted input to an action. How many output did the CNN have? How did they train the neural network for prediction?
Here are the steps that I believe are happening inside DQN:
1) A game picture (a state) is send to CNN as input value
2) CNN predicts an output as action (eg:left, right, shoot, etc)
3) Simulator applies the predicted action and moves to new game state
4) repeat step 1
The problem with my above logic is in step 2. CNN is used for predicting an action, but when is CNN trained for prediction?
I would prefer if you used less math for explanation.
EDIT
I want to add some more questions regarding the same topic
1) How reward is passed in the neural network? that is how neural network knows whether its output action obtained positive or negative reward?
2) How many output the neural network has and how action is determined from those outputs?