I implemented Actor-Critic with N-step TD prediction to learn to play 2048 (link to the game : http://2048game.com/)
For the enviroment I don't use this 2048 implementation. I use a simple one without any graphical interface, just pure matrices. The input for the neural network is the log2 of the game board.

The structure of my network is:
1. Input layer
2. Hidden layer with 16 units
3. Softmax layer with 4 units (up, down, left, right) for the actor
4. Linear regression for the critic
The hidden layer is shared between the third and fourth layer.

The reward in the orginal game is the value of the merged cells. For example, if two fours merged than the reward is eight. My reward function is almost the same, except I take the log2 of it. I tried these parameteres and I also tweaked the learning rate, the gamma, but I couldn't achive any good result.

Could You recommend what should I change?


Interesting project. First thing I'd do is normalize your state by the maximum cell value. This way you can represent multiple situations at once (eg A grid of all 4s and 8s would look the same as a grid of all 16s and 32s). Also making the reward = max_cell/2048 might do better as ActorCritic methods seem to do better with rewards within 0-1.

Another reward setup is giving +1 per timestep. It's simple but it also means that maximizing the time to stay alive is the best, which is what I end up doing most of the time when I play anyway.

Good luck!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.