How does deepmind's Atari game AI work?

I know that deepmind used deep Q learning (DQN) for its Atari game AI. It used a conv neural network (CNN) to approximate Q(s,a) from pixels instead of from a Q-table. I want to know how DQN converted input to an action. How many output did the CNN have? How did they train the neural network for prediction?

Here are the steps that I believe are happening inside DQN:

1) A game picture (a state) is send to CNN as input value

2) CNN predicts an output as action (eg:left, right, shoot, etc)

3) Simulator applies the predicted action and moves to new game state

4) repeat step 1

The problem with my above logic is in step 2. CNN is used for predicting an action, but when is CNN trained for prediction?

I would prefer if you used less math for explanation.

EDIT

I want to add some more questions regarding the same topic

1) How reward is passed in the neural network? that is how neural network knows whether its output action obtained positive or negative reward?

2) How many output the neural network has and how action is determined from those outputs?

• We prefer only one question per post. If you have more questions, please ask them in a different post. – Mithical Sep 11 '16 at 10:44

Training happens once you have a result. If the result is good (maybe you won in pong, or you improved your highscore in breakout) all the actions in the game are "supported" by backpropagation, if the result is bad, all the actions in the game are suppressed.

This sounds weird because in each game regardless of the end result you'll have many good and bad actions, but it works if you keep it up for many thousands of games.