I'm about to create an OpenAI Gym environment for a flight simulator. I'm wondering, how to cope with the fact, that the result and reward for some action needs a considerable time to advance through the system due to the inherent time constants.
In the easy example Gym-environments (e. g. cartpole, or some games) the step can anstantly be executed and the resulting reard can be calculated.
In my continuous control system (aka flight simulator), There is some reaction time needed, until I can calculate the result from my action. E. g. When I pull the stick, it takes some time, until the aircraft lifts it's nose. So there is a considerable delay (maybe in the seconds ballpark) from commanding the action to the environment, and the earliest observation of that result and it's corresponding reward.
How can I cope with that. As far as I understood, the
env.step(action) function blocks till it comes back with a new observation and a corresponding reward.
- How can I cope with long lasting reward caclulations?
- Is it possible to have overlapping actions somehow? E. g. command a new action every 100ms, but get the reward for that action only 1 second later. In this case there would be always 10 rewards pending.
I hope I made my point clear. Don't hesitate to ask for further details in the comments.
Any hints are welcome. Is there anything to read out in the wild dealing with a similar issue?