2
$\begingroup$

I am planning to use OpenAI gym for my experiment in real life. In my experiment design, by the limits of a real-life scenario, I can only receive the state information or the rewards about 2-3 timesteps behind when the action has happened (in OpenAI gym term, ~3 cycles of step(action) function has occurred). For example, by the time the state at timestep i is observed, an action at timestep i+3 would have happened.

From how I perceive the function, step(action), is that it needs to return next_state, reward, done every step. And the agent will learn from state -> action -> next state -> reward tuple. So I was wondering if can I cache the action for future use along with the state with the correct time step in OpenAI gym? or delay the state observation/reward instead? Could the OpenAI be able to learn?

I am experimenting with PPO TD3 SAC which all uses actor-critic networks. Would the network eventually be trained well enough to the point where it would still perform well with the delayed state observation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

To turn this delayed action approach into a normal, and theoretically valid MDP, add a "pending action resolution" array to the state representation, and ensure that the state transition (or step code) manages this array with each new action pushed into it, and the array shifting down as it goes. The array length should be minimum of the number of timesteps required to fully resolve a previous action.

This allows your agent to see the actions it has recently performed and account for them in calculations without any adjustment needed to the core MDP formulation or standard agent designs.

If delays are stochastic or vary (e.g. a scenario where a warehouse is ordering stock), then add more data to this pending action resolutions array as necessary, such as expected resolution time, or time since action was taken.

Improve this answer
$\endgroup$
5
  • $\begingroup$ From what I understand in implementation of your 'pending action resolution' array is: First-In the step(action), take an action in real life. Second- Buffer/queue the action into the array with length of the delay step. Third - repeat fist&second until the action resolution array is completely filled. Last - With the full array, measure the state + reward associated with the oldest/last index of the pending action resolution array. Is this correct? If so, what do I return at the end of the step() function in the Second if I do not have the next state observed and the reward? $\endgroup$
    – Ykwk
    Sep 3, 2021 at 20:40
  • $\begingroup$ @Ykwk: That's overcomplicated. The array of recent actions becomes just a normal part of the state. Initialise it with "null" actions so it has a fixed length. When the agent takes an action, push the id of that action onto the array, then shift the array to remove one old action (so the array stays the same length, and actions are stored in age order). Run all the rest of MDP and time step handling as normal for the agent - i.e. process value estimates, use replay memory for training etc on every time step. $\endgroup$
    – Neil Slater
    Sep 3, 2021 at 20:50
  • $\begingroup$ How can the 'initialization null' array of action be implemented, when every iteration of step() requires to return -next state -reward? Passing in 'null' action does not give me any return values for step() function. I don't think I can get the next action, without having the agent to go to next state reward. $\endgroup$
    – Ykwk
    Sep 7, 2021 at 3:32
  • $\begingroup$ @Ykwk These are not actions that are taken in any sense, they are placeholders for "no action" in the state representation so that the feature size is consistent, so that you can use a neural network. Another term might be "padding". Implement them as part of the starting state representation. $\endgroup$
    – Neil Slater
    Sep 7, 2021 at 6:16
  • $\begingroup$ Can you share minimal working example code @Ykwk $\endgroup$
    – Muhammad Yasirroni
    Jun 28, 2022 at 8:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .