1
$\begingroup$

I'm about to create an OpenAI Gym environment for a flight simulator. I'm wondering, how to cope with the fact, that the result and reward for some action needs a considerable time to advance through the system due to the inherent time constants.

In the easy example Gym-environments (e. g. cartpole, or some games) the step can anstantly be executed and the resulting reard can be calculated.

In my continuous control system (aka flight simulator), There is some reaction time needed, until I can calculate the result from my action. E. g. When I pull the stick, it takes some time, until the aircraft lifts it's nose. So there is a considerable delay (maybe in the seconds ballpark) from commanding the action to the environment, and the earliest observation of that result and it's corresponding reward.

How can I cope with that. As far as I understood, the env.step(action) function blocks till it comes back with a new observation and a corresponding reward.

  • How can I cope with long lasting reward caclulations?
  • Is it possible to have overlapping actions somehow? E. g. command a new action every 100ms, but get the reward for that action only 1 second later. In this case there would be always 10 rewards pending.

I hope I made my point clear. Don't hesitate to ask for further details in the comments.

Any hints are welcome. Is there anything to read out in the wild dealing with a similar issue?

Cheers,

Felix

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It's not on your end, as a creator of flight simulator, to worry about what action should get the credit for the reward that happened some time after the action was taken. You should return the reward when the actual event happens not when the action that caused it happened. It's the job of the reinforcement learning agent to figure that out. For example if you want to give a reward when airplane nose is at 45 degrees from the horizontal axis, you should return the reward when that event actually happens, RL agent should figure out that the crucial action happened some time ago. This may be difficult for the agent but its up to the user to use a proper algorithm and proper exploration strategy to solve the problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think you got me wrong: I'm not the creator of the flight simulator, but the creator of the agent and of the gym-interface. What you said in the second half of your response is the interesting part for me: When I get you right, you propose, that the agent sees the reward corresponding to the current situation, no matter when the action happend, that led to this situation. So the delay between action and reward shall be figured out by a proper algorithm in the agent. Will we need some special treatment in the agent, or will this turn out in the course of learning? $\endgroup$
    – opt12
    May 27, 2019 at 15:32
  • 1
    $\begingroup$ Strong feature of RL is that it deals well with temporal credit assignment problem, in general you shouldn't need any specialized techniques, standard deep RL algorithms like PPO, ACER, DQN, etc. should be able to work fine, some better, some worse, you should try it out. Your case isn't unique, currently in OpenAI Gym there are environments with same challenges, especially Atari games. For example Pong, you get a reward when you score against opponent, but the action that "scored" happened when you touched the ball with paddle and between those 2 moments several frames and actions happened. $\endgroup$
    – Brale
    May 27, 2019 at 15:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .