3
$\begingroup$

I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects the agent's behavior at timestep 1 and is not used at any of the later timesteps.

To illustrate my specific case, here is an image of my enviroment:

enter image description here

At step 0 red ball recives the action (force at a certain angle) and starts moving. At each timestep the coordinates of the red and blue ball are computed, but we cannot interfere with the enviroment anymore (the final configuration is already determined by the initail impulse and the physics).

Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).

I am new to reinforcement learning so I apologize if this is a silly question

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).

Yes from the perspective of the agent this is a single-step problem. As such, it is not a full reinforcement learning problem, but can also be considered a contextual bandit problem.

The time steps of the simulation are not relevant to the agent if it does not get to take actions for them. They are external to the agent, and an implementation detail of how you calculate the outcome of its action.

You can still treat this as a full RL problem where the agent has one available "do nothing" action for later steps. You might want to do that in case you would like the agent code to predict the outcome as it observes later states. This may even improve the performance of the agent, as all those observations will allow it to learn physics rules that it might be able to generalise from when considering a starting state. For this to work, you may need to encode more details in the state though, because a static snapshot of positions is not enough data to predict what will happen next.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Does using a "do nothing" action result in sparse rewards, potentially slowing learning? $\endgroup$
    – Luca Anzalone
    Jun 18 at 12:40
  • 1
    $\begingroup$ @LucaAnzalone that depends entirely on how rewards are calculated, and you have to balance any negative impact from that against the possibilities of easier learning of the physics. I don't think it's easy to predict without trying it $\endgroup$
    – Neil Slater
    Jun 18 at 14:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .