3
$\begingroup$

I was wondering how can we deal with movement inertia in an environment that is constantly changing?

Imagine that you make a step on an environment that moves a ball. When you make the step, you make the ball move and at one time, it returns an observation and a reward. Then you give the observation and the reward to your RL algorithm and you find the next best policy.

While calculating the best policy, the ball is still moving in the environment because it has an inertia made by the action we took previously. How do we deal with that? The policy that we just calculated is already based on an old value right?

If the RL algorithm and the communication between the RL and the environment is fast I don't think it is much of an issue, but what if your algorithm is very slow ?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ You could also consider this in terms of predictive control rather than a pure RL task. $\endgroup$
    – Bruce Adams
    Commented Sep 4, 2023 at 11:58

2 Answers 2

Reset to default
6
$\begingroup$

The simplest answer is that the inertia (or velocity, if the mass of the ball is not a variable) should be part of the observable state that the agent has access to. In RL there are usually two parts to outcome from actions - one is any immediate reward, and the other is changes to state that may impact future actions and rewards. Provided the agent can track in its state representation all factors that could impact future rewards, then most RL algorithms will work as intended (some will be better than others depending on what parts of the problem are hard to solve).

A common hack to representing velocity is to "frame stack" multiple static observations as the current state. This was used for example in the original paper that used DQN to solve Atari games.

If the RL algorithm and the communication between the RL and the environment is fast I don't think it is much of an issue, but what if your algorithm is very slow ?

The basic MDP model used in RL is designed around a situation where you have some control over what the time steps are. If training is done in a real physical environment, and you are not able to semi-reliably sample observations and act with a fixed cadence (e.g. ten times per second), then you may have problems. A lot of game-based RL ignores this issue, runs faster than real time anyway, and only allows the game emulator to move forward in time when it is ready to act and collect more training data.

If you really are in a real time environment, outside of a simulator, where events happen too fast to reliably train, then you may have issues. One possible solution to look into is real-time RL (link chosen randomly, but hopefully gives you some leads) where time steps can be more flexible, although if your agent is really slow compared to the use case, results will be limited by response time - potentially making it infeasible to make it optimal. You can also use an agent that only implements a current behaviour policy, and run training updates based on results asynchronously on a separate device, so you would not be limited by training speed, only by inference speed that a production-ready agent would be restricted by too.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot for you answer. I am using the velocity of the ball as an observation state and it's position. I think to start, I will try couple of model and test it to see if they are fast enough to reduce the time difference at a point that it can be ignored. I will definitely take a look at the real-time RL by curiosity. For info, I'm not using frame and I have control on the environment but I would like to avoid making a step and stopping the environment in order to make all the prediction. If I see that it's not possible, I will have to change my mind and stop the environment. $\endgroup$
    – CyDevos
    Commented Sep 4, 2023 at 14:27
  • $\begingroup$ @CyDevos If your inference time is 1ms (time between collecting observation and triggering next phase of action), and making samples and action decisions once every 100ms is adequate, then your model will probably adjust to not having super-precise measurements, and you can set time steps at 100ms and not worry too much. Even having observations that are significant fraction of time step out of date may not be a major issue, and the agent should be able to adjust if this is a consistent time offset - however, at some point the real time model may be better $\endgroup$
    – Neil Slater
    Commented Sep 4, 2023 at 15:45
  • $\begingroup$ I will try to get a timing estimation on my environment. I will definitly check that real time model at one time. The paper is very interesting btw. I will let you know how everything is going :) $\endgroup$
    – CyDevos
    Commented Sep 5, 2023 at 8:13
  • 1
    $\begingroup$ Hi, @Neil Slater, indeed the timing between the environment and the RL has no effect as the prediction is really quick. The agent still learned and successfully completed the task. $\endgroup$
    – CyDevos
    Commented Sep 23, 2023 at 10:19
4
$\begingroup$

In RL usually a "discretized" time MDP is used in case of continuous time problems, with the idea that if we evaluate enough time the policy, we can approximate it the continuous case. This is also motivated by the fact that we have no idea how to handle continuous case scenarios even software-wise, as even videos are just a fast-enough sequence of images.

At this point, if your policy is fast enough (is not a LLM that takes minutes to generate an answer), you can just go and evaluate the policy at each timestep (if images, the video framerate at most)

You can make a point about the fact that during training you can wait the policy to evaluate and then pass to the next frame, where in real life you cannot, but at that point it's on you to develop a faster model.

In all the cases, you can at the beginning develop a model, test it in deployment to check it's performance, observe how much it takes to make a decision, and use that in the environment to improve the training.

About the inertia, you have another problem, which is that the first moment cannot be estimated with a "single point", so you'll have to provide your model at least the previous state so that it's able to estimate it

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot for your answer. It is very clear. I will try to develop and test different model and see which one fit the best my use cases. $\endgroup$
    – CyDevos
    Commented Sep 4, 2023 at 14:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .