I am planning to use OpenAI gym for my experiment in real life.
In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 timesteps behind when the action has happened (in OpenAI gym term, ~3 cycles of
step(action) function has occurred). For example, by the time the state at timestep i is observed, an action at timestep i+3 would have happened.
From how I perceive the function,
step(action), is that it needs to return
next_state, reward, done every step. And the agent will learn from state -> action -> next state -> reward tuple. So I was wondering if can I cache the action for future use along with the state with the correct time step in OpenAI gym? or delay the state observation/reward instead? Could the OpenAI be able to learn?
I am experimenting with PPO TD3 SAC which all uses actor-critic networks. Would the network eventually be trained well enough to the point where it would still perform well with the delayed state observation?