# Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all rewards are negative) I tested is of type $$R(s, a, s')$$. During training, it can happen that not only one but several actions are applied simulataneously to the environement before a reward is returned:

$$s_t →a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}$$ instead of $$s_t →a_t→s_{t+1}$$.

Out of all actions applied, only one is generated by the agent. The others are either a copy of the agent's action or are new values.

If I look at the tensorboard output of the trained agent, it looks rather horrific as displayed below (~ zero explained variance, key trainig values do not converge or behave weirdly, etc. etc.).

Obviously, the training did not really work. Now I wonder what the reason for that is.

1. Is it possible to train an agent using a reward function of type $$R(s, a, s')$$ even if several actions are applied simulataneously or is this not possible at all? Other agents I trained using a reward function of type $$R(s,a)$$ have a better tensorboard output so I guess that this is the problem.
2. Or is maybe another reason more likely to be the root of the problem? Like a bad observation space formulation or hyperparameter selection (both for RL algorithm and reward function used).  • You could always change things to satisfy the usual convention $s_t→a_t→s_{t+1}$. You could either have an action vector $\mathbf a_t=(a_{t1},a_{t2},a_{t3})$ that includes all intermediate actions and then at the output have nodes for all possible action triplets or you could introduce intermediate states $s_t→a_{t1}→s_{t1}→a_{t2}→s_{t2}→a_{t3}→s_{t+1}$. Also what does it mean that reward function is formulated as a loss function ? – Brale_ Aug 7 at 14:26
• I think the term you're looking for is that reward function is negative definite function, meaning, for all function inputs, output is negative. As for the actions, the fact that only one action is generated by the agent means that those other actions shouldn't be treated as actions but rather state features. Consider this sequence: $s_t→a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}→a_{t+1,1}...$. Let's say actions $a_{t,1}$ and $a_{t+1,1}$ are generated by the agent. Then, the state features for generating action $a_{t+1,1}$ should be $\mathbf s'_{t+1}=(a_{t,2}, a_{t,3},s_{t+1})$ – Brale_ Aug 7 at 15:33