In a classic GridWorld Environment where the possible actions of an agent are (Up, Down, Left, Right), can another potential output of Action be "x amount of steps" where the agent takes 2,3,.. steps in the direction (U,D,L,R) that it chooses? If so, how would one go about doing it?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
You can definitely define an environment that accepts more types of action, including actions that take multiple steps in a direction.
The first thing you would need to do is implement support for that action in the environment. That is not really a reinforcement learning issue, but like implementing the rules of a board game. You will need to decide things such as what happens if the move would be blocked - does the move succeed up the point of being blocked, does it fail completely, is the reward lower depending on how much the agent tries to overshoot, etc.
After you do that, you will want to write an agent that can choose the new actions. You have a few choices here:
Simplest would be to enumerate all the choices separately and continue to use the same kind of agent as you already have. So instead of $\{U, D, L, R\}$ you might have $\{U1, U2, U3, D1, D2, D3, L1, L2, L3, R1, R2, R3\}$.
If you want to take advantage of generalisation between similar actions (e.g. that action $U3$ is similar to $U2$ and also to $R3$), then you can use some form of coding for the action, such as the relative x,y movement that it is attempting. So you could express $U2$ as $(0,2)$ and $L3$ as $(-3,0)$. For that to then work with Q values, you cannot easily use a table. Instead, you would need to use function approximation, for example a neural network, so you can implement $q(s,a)$ as a parametric function - combining $s,a$ into the input vector, and learn the parameters to that the neural network outputs the correct action value. This is what the Q learning variation DQN can do, as well as other similar RL algorithms that use neural networks.
Using a neural network, instead of tabular Q-learning, is not something you see often with grid world environments. It is a step up in complexity, but it is often required if state space or action space becomes large and might benefit from the generalisation possible from trainable function approximators.
answered Jun 16, 2020 at 20:43
-
$\begingroup$ @ NeilSlater Ah yes I expected that I would have to do something like the first bullet point. Yes, I saw in different videos and tutorials that tabular Q-learning is better when dealing with GridWorld environments. So would it not be possible for an environment itself to decide how many steps it should take after choosing a direction to take? Additionally, could you please explain the second bullet point in a bit more detail, specifically the first sentence of it? Thank You! $\endgroup$ Jun 16, 2020 at 21:19
-
$\begingroup$ @HuzaifahShamim I added a couple of examples to the second bullet point $\endgroup$ Jun 16, 2020 at 21:25
-
$\begingroup$ @ NeilSlater, how is expressing U2 and L3 that way different then if I was to do it via QTable? I was thinking that that is how I would express it if I was to do it like the first bullet point. $\endgroup$ Jun 16, 2020 at 22:12
-
$\begingroup$ @HuzaifahShamim Well you can always enumerate and put into a table, and/or one-hot encode if you wish. But the numerical coordinates version can also be used as input into an approximation algorithm, supporting generalisation - something you generally want when the combined state and action space becomes large. $\endgroup$ Jun 16, 2020 at 22:41
-
$\begingroup$ There is no requirement to try second bullet point, if you don't feel you need it. As explained just increasing the enumerations is simplest and just fine up to the point when you have too many options to handle. 12 or 16 or 20 action choices is not many. But 2000 might be too much if the agent could choose to move anything from 1 to 500 steps in any direction in a single time step $\endgroup$ Jun 16, 2020 at 22:44