I try to implement RL to a case something like this:

This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different available rooms with different properties. At the beginning of a round, each player will be given 10 rooms one-by-one (the sequence is same for each player) from 1000 that available, then he/she try to create a maze from the taken rooms (by arranging each room). After the maze is done there is a Game Master who will judge the maze (give a score between 0-100). The player never know how the Game Master judges the maze, it can be judged based on the level of difficulty that is produced, the order of the room we compose, or others. The player who got the best score for the given rooms will win this round.

in this case, I have around 100,000 "perfect" mazes that have been created from different room combination and got a perfect score. I use this maze as episodes and try to train RL-Agent to find the pattern of how the Game Master judges a maze. For your information, there are rooms that not exist in the 100,000 perfect mazes, but I hope the RL-Agent can use its properties to find similar rooms that exist in the "perfect" mazes, and make it as a reference

This case is different from other RL environments that I've ever met before, generating an episode is not an easy task because it needs an expert to validate it (The Game Master). So you could say, I can only build the RL agent using that 100,000 episodes.

But, even though it only consists of 100,000 episodes, my case has millions of states, so I plan to use Q-Learning with Neural Net as approximator.

My question is:

  • in this case, am I still need the Experience Replay process (I am afraid I don't need it because of the small number of available episodes)?
  • has this case ever happened before? What is the best approach to deal with cases where the number of episodes is limited?
  • $\begingroup$ Is your goal to find a good policy for the environment? How will you test your new policy? $\endgroup$ – Neil Slater Mar 26 '19 at 11:04
  • $\begingroup$ @Neil Slater I'll test the new policy by test it to the available episodes that unused before (like "train-test" split concept). But it's just for the evaluation in the final phase $\endgroup$ – malioboro Mar 27 '19 at 8:23
  • $\begingroup$ That may not work very well, depending on what data you hold. But not important to this question - perhaps you can ask another about that. I was asking to try and get a sense for how you might use the data. I have a better clarifying question now: $\endgroup$ – Neil Slater Mar 27 '19 at 9:31
  • $\begingroup$ What kind of behaviour do your 100,000 pre-defined episodes represent? Is it mainly an "expert policy" which attempts to be optimal, or is it more varied than that? Could you give a little bit of detail on how the 100,000 "sequences" were generated? Please edit into the question. That could help guide an answer. An important detail would be if the sequences were generated using a known policy for instance (because then you could use importance sampling) $\endgroup$ – Neil Slater Mar 27 '19 at 9:33
  • $\begingroup$ @Neil Slater I've updated my question $\endgroup$ – malioboro May 3 '19 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.