I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all rewards are negative) I tested is of type $R(s, a, s')$. During training, it can happen that not only one but several actions are applied simulataneously to the environement before a reward is returned:

$s_t →a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}$ instead of $s_t →a_t→s_{t+1}$.

Out of all actions applied, only one is generated by the agent. The others are either a copy of the agent's action or are new values.

If I look at the tensorboard output of the trained agent, it looks rather horrific as displayed below (~ zero explained variance, key trainig values do not converge or behave weirdly, etc. etc.).

Obviously, the training did not really work. Now I wonder what the reason for that is.

  1. Is it possible to train an agent using a reward function of type $R(s, a, s')$ even if several actions are applied simulataneously or is this not possible at all? Other agents I trained using a reward function of type $R(s,a)$ have a better tensorboard output so I guess that this is the problem.
  2. Or is maybe another reason more likely to be the root of the problem? Like a bad observation space formulation or hyperparameter selection (both for RL algorithm and reward function used).

Thanks for your help!

enter image description here

enter image description here

  • $\begingroup$ You could always change things to satisfy the usual convention $s_t→a_t→s_{t+1}$. You could either have an action vector $\mathbf a_t=(a_{t1},a_{t2},a_{t3})$ that includes all intermediate actions and then at the output have nodes for all possible action triplets or you could introduce intermediate states $s_t→a_{t1}→s_{t1}→a_{t2}→s_{t2}→a_{t3}→s_{t+1}$. Also what does it mean that reward function is formulated as a loss function ? $\endgroup$
    – Brale
    Aug 7 '19 at 14:26
  • $\begingroup$ Hey, thanks for your answer. The suggestion using intermediate steps sounds quite interesting, however, it probably is not applicable to my environment since it requires all steps to be performed simultaneously. Or at least the reward function requires it. But I'll look some more into it. To specifiy the actions applied: only one of them is generated by the agent. The others are not. With the term loss function I want to imply that all rewards are negative. $\endgroup$
    – mglss
    Aug 7 '19 at 15:10
  • 2
    $\begingroup$ I think the term you're looking for is that reward function is negative definite function, meaning, for all function inputs, output is negative. As for the actions, the fact that only one action is generated by the agent means that those other actions shouldn't be treated as actions but rather state features. Consider this sequence: $s_t→a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}→a_{t+1,1}...$. Let's say actions $a_{t,1}$ and $a_{t+1,1}$ are generated by the agent. Then, the state features for generating action $a_{t+1,1}$ should be $\mathbf s'_{t+1}=(a_{t,2}, a_{t,3},s_{t+1})$ $\endgroup$
    – Brale
    Aug 7 '19 at 15:33
  • $\begingroup$ Thanks for your input!. It helped me a lot finding out about how to solve my problem. In the end, the problem in my specific case was the formulation of the reward function and the corresponding state space formlation. $\endgroup$
    – mglss
    Aug 8 '19 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.