I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
- In a 5 by 5 board, each position can contain a card with a color (0-4) and a value (0-9).
- Some initial cards, all distinct from each other, may be on the board.
- In each round, a card not on the board is chosen with uniform distribution.
- The player has to place this card in an empty position before the next card is chosen.
- When all positions are filled, the game ends and the final score is the sum of all points of each row, each column, and each of the two diagonals, where the points are calculated as follows (order does not matter, e.g. AAABB=ABABA, 12345=23415):
|Not all identical
|Not all consecutive
|Five of a kind
|Four of a kind
|Three of a kind
The goal is of course to get a score as high as possible.
Question: How should I write the reward function to teach the agent the rules?
I used invalid action masking, so no invalid actions can be taken. However, it's still hard for high-score combinations to randomly show up. I tried setting reward to be the points possibly added due to the placed card (like if the placed card can form a pair with another one in the same row, add some points based on how much point a pair is rewarded in the final score), and reward 100*(final score) at the end, hoping the agent to learn that the final score is the most important, but the mean reward was not improving even after 2 million timesteps.
I use MaskablePPO with MlpPolicy (alias of MaskableActorCritic) from sb3-contrib. I represent each card as an int8. The upper 4 bits representing the color and the lower 4 bits represent the value.
The observation space I use consists of the card to be placed, the cards on the board, and all cards not on the board. I'm wondering if I should include cards not present in the observation space. I tried to exclude it but it does not change the results much.
Apart from direct answers, any pointers/reference would be much appreciated.