I try to implement RL to a case something like this:
This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different available rooms with different properties. At the beginning of a round, each player will be given 10 rooms one-by-one (the sequence is same for each player) from 1000 that available, then he/she try to create a maze from the taken rooms (by arranging each room). After the maze is done there is a Game Master who will judge the maze (give a score between 0-100). The player never know how the Game Master judges the maze, it can be judged based on the level of difficulty that is produced, the order of the room we compose, or others. The player who got the best score for the given rooms will win this round.
in this case, I have around 100,000 "perfect" mazes that have been created from different room combination and got a perfect score. I use this maze as episodes and try to train RL-Agent to find the pattern of how the Game Master judges a maze. For your information, there are rooms that not exist in the 100,000 perfect mazes, but I hope the RL-Agent can use its properties to find similar rooms that exist in the "perfect" mazes, and make it as a reference
This case is different from other RL environments that I've ever met before, generating an episode is not an easy task because it needs an expert to validate it (The Game Master). So you could say, I can only build the RL agent using that 100,000 episodes.
But, even though it only consists of 100,000 episodes, my case has millions of states, so I plan to use Q-Learning with Neural Net as approximator.
My question is:
- in this case, am I still need the Experience Replay process (I am afraid I don't need it because of the small number of available episodes)?
- has this case ever happened before? What is the best approach to deal with cases where the number of episodes is limited?