0
$\begingroup$

I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that the goal, obstacles, and player will be distributed differently each time.

When testing the learning on the randomly initialized grid, the simpler network - a vanilla one - is able to win roughly 50% of the time. When an 'experience replay' mechanism is added, it attains more than 90% success. My question is how the network is able to achieve these results, given that - for a given state - the optimal action will be different every time the environment is initialized.

You can find the full code in

DRL in action Chap 3 github page

$\endgroup$
1
  • $\begingroup$ I was wrong. It is not true that for a given state the optimal action will be different every time the environment is initialized. $\endgroup$ Commented Aug 8, 2023 at 14:36

1 Answer 1

1
$\begingroup$

Thanks to the power of generalization of neural networks... the whole success of those models is their ability to be extremely effective in generalizing in high dimensions

So, during training, the network is not learning the usual $|S|\times|A|$ Q-learning table, instead it's learning a compressed version of it, considering similarity between states

Think about snake but with walls... instead of learning that in that position there is a wall, it learns that if it sees the head near a wall, it has to go in a direction that allows it not to smash into that (generalize)

Obviously, this generalization, is only achieved if it's thought to the network, thus if during training, you also train it in different environments

$\endgroup$
3
  • $\begingroup$ Thank you Alberto. I have not realized, and is a crucial aspect, that the agent (neural network) knows the different environment structures because it is encoded in the state vector. Knowing that, each state has one, and only one optimal action, and so is not so surprising that the network learns to act optimally. One could also do it as a look up table in the case of this example, because the number of different states is not so large. You are right that the network should be trained on the environment "random" mode. Else, it should not learn to cope with this situation. $\endgroup$ Commented Aug 8, 2023 at 14:33
  • $\begingroup$ In the Zai&Brown example, the grid is 4 by 4. As explained in page 84, "The number of possible game configurations (the size of the state-space) is approximately 16*15*14*13 =43680, since there are 16 possible positions the agent can be in on a 4*4 grid, and then 15 possible configurations for the wall, since tha agent and wall can´t be overlapping in space, etc.)". For a 5*5 grid we should have 25*24*23*22 = 303600 different configurations. No so much. $\endgroup$ Commented Aug 8, 2023 at 14:51
  • $\begingroup$ @HermesMorales you know what? I'm so used to look at functions, that I forgot how to do multiplications... however, we agree that the more the walls, the more the states, and If im not mistaken again, with more than 5 walls, get's pretty large, so you have to rely on generalization $\endgroup$
    – Alberto
    Commented Aug 8, 2023 at 14:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .