Return to Answer

extended answer based on the comments

Source Link

edited Dec 14, 2018 at 22:03

50k4

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.

Source Link

created Dec 14, 2018 at 13:13

50k4

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.