3
$\begingroup$

Problem description:

Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:

  • if current action == previous action, you get reward = $R(a,s)$
  • if current action != previous action, you get reward = $R(a,s) - \text{penalty}$

In this environment, switching actions bears a significant cost. We would like the RL algorithm to learn optimal actions under the constraint that switching action is costly, i.e. we would like to stay in selected action as long as possible.

The penalty is significantly higher than an immediate reward, so if we do not take it into account, the model evaluation will have a negative total reward with almost 100% probability, since the agent will be constantly switching and extracting rewards from environment that are smaller than the cost of switching actions.

Action space is small (2 actions: left, right). I'm trying to beat this game with PPO (Proximal Policy Optimization)

Questions

  • How one might address this constraint: i.e. explicitly make the agent learn that switching is costly and it's worth sitting in one action even if immediate rewards are negative?

  • How can you make the RL algorithm learn that it's not the reward term $R(a_t|s_t)$ that is negative, and thus decreasing $Q(a_t|s_t)$ and $V(s_t)$, but it's the penalty term (taking the action that is different from the previous action at step $t-1$) that is pushing total reward down?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
3
$\begingroup$

The answer to both your concerns is:

  • Add the previous action choice to the state representation.

It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.

By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, which you had otherwise lost by making the reward dependent on a variable that was both systematically changing and hidden from the agent.

The state in a MDP is often not just the current observations that the environment provides, but can include any relevant knowledge that the agent has. At the extreme that can include a complete history of all observations and actions taken to date. It is common practice to derive the state as a summary of recent history of observations and actions taken so far. In your case, all you need do is concatenate the previous action to the observation, because you know about the constraint and how it affects optimisation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! That was my initial idea too. You are absolutely right, that MDP property has to be restored. My concern is that adding previous action to the state representation might be not enough, as it will take some time for the agent to learn to associate this state feature with reward. Maybe there is a more explicit way of influencing learning that switching actions bears a penalty? $\endgroup$
    – FQT
    Commented Jan 26, 2021 at 10:18
  • $\begingroup$ @FQT: There is no specific clever way that is normally used in RL to solve this issue as far as I know. However, the new MDP with the previous action might be solved faster using a different algorithm, as that part of the behaviour is deterministic. You could try to re-structure the agent in other ways too, because you know specific things aboutthis penalty that you might not know about the rest of the environment (for instance reward prediction models could focus only on the other part of the environment with action matching hard-coded), but modifying state is simpler and IMO best. $\endgroup$
    – Neil Slater
    Commented Jan 26, 2021 at 10:25
0
$\begingroup$

As said, you will have to include previous input state(s) in your training input patterns.

Suppose we use straightforward NN backpropagation learning..

You would expand the input layer with additional neurons and weights connected to the past. The time window neurons should introduce additional weights into the first hidden layer. A neural net architecture doing this is called Time Delay neural net (TDNN)

https://stats.stackexchange.com/questions/160070/difference-between-time-delayed-neural-networks-and-recurrent-neural-networks

I used TDNN in the past for signal processing. My TDNN includes the past of N inputs, supporting M steps in the past. A flexible TDNN can also be configured to connect some hidden layer past output to the next hidden layer.

https://neuron.eng.wayne.edu/tarek/MITbook/chap5/5_4.html

https://www.mathworks.com/help/deeplearning/ref/timedelaynet.html

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .