I want to train a feedforward neural network to play a video game called Puyo Puyo 2, using reinforcement learning. More specifically, I'm trying Q-learning but I'm open to better alternatives.
In this video game, you start with an empty 12x6 board which you fill with falling pairs of colored blobs called puyos. When 4 or more puyos of the same color are connected horizontally and/or vertically, they disappear and the puyos above them fall down, potentially creating new groups of puyos that disappear and so on, creating a chain (here is a chain example : https://puyonexus.com/chainsim/image/5SoqU.gif). My goal is to teach the neural network to build the longest chains possible given a random sequence of puyo pairs. The game's information is incomplete though, as you only see the current and the next two pairs at a given time.
What is the best choice for the output layer: a layer with 22 neurons, one for each possible way to place a pair at a given time, and a softmax activation function, or a layer with a single output node, with a linear activation function? Or something else? Also what do you think is a good number of hidden layers and neurons per layer and what activation function should I use for the hidden layers?
What should my target Q-value be when I evaluate a move, so that I can train the network on each game situation? Right now I'm thinking about using the maximum chain possible after the move is done, divided by half of the number of moves played so far. This is because if you play optimally and you are lucky enough, you can increase your maximum chain by 1 every 4 puyos you place, that is every two moves. Thus my Q-value would stay between 0 and 1. What is the best update formula I can use? Is the one on wikipedia correct?
If I use a softmax output layer, how do I update the target Q-value vector? The update formula gives me an update but then the vector doesn't represent probabilities anymore. Can I add the update then take the softmax of the vector again just to make it probabilities again?