Reinforcement Learning with Adaptive Action Magnitude

Question

How to select an action in a state if the action does not necesarily cause the environment to change state?

Given 10 states ($S_0$ to $S_9$) and in each state $i$ there are two actions defined $(1,-1)$. $1$ increases a parameter of the environment and $-1$ decreases it.

For example, if the parameter is speed and it is currently 1 rad/s, corresponding to ${S_i}$, an action can be either increase or decrease this and therefore transition to the next state, ideally $S_{i-1}$ or $S_{i+1}$.

It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action.

The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change.

For example, if the envitonment is instate $S_1$ and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state $S_2$? How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.

For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and my Q table is $10 \times 2$.

Answer 1

There are more possible approaches to tackle this problem:

1. Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.

2. Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.

3. You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.