I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to test this algorithm assumes R as a function of only states.

My question is:

Given $\mathcal{X}$ as the set of states of an MDP and $\mathcal{A}$ as the set of actions of an MDP. Supposing I have four states ($1$,$2$,$3$,$4$), two actions $a$ and $b$ and a reward function $R: \mathcal{X}\to\mathbb{R}$ s.t.

$R(1) = 0$

$R(2) = 0$

$R(3) = 0$

$R(4) = 1$

If I need to change the current reward function to a new reward function $R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$ is it ok to compute it as $\forall a,R(s,a) = R(s)$?

$R(1,a) = 0$

$R(1,b) = 0$

$R(2,a) = 0$

$R(2,b) = 0$

$R(3,a) = 0$

$R(3,b) = 0$

$R(4,a) = 1$

$R(4,b) = 1$

More generally, what's the correct way of generalising a reward function $R: \mathcal{X}\to\mathbb{R}$ to a reward function $R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$?


1 Answer 1


As explained here, I can write $R(s,a) = R(s)\ \forall a$ since the reward of my specific MDP is dependent exclusively to the state $s$.

  • $\begingroup$ Just a clarification, if you're trying to reproduce a certain paper that uses $R(s, a)$, why do you say that in your MDP the reward function only depends on $s$? I suppose you're testing their approach in a simpler environment. Right? Is the code you're using available? $\endgroup$
    – nbro
    Jan 31, 2021 at 19:04
  • $\begingroup$ Yes exactly, since the paper doesn't present an example environment, I'm using the one from here, the code isn't available yet (I couldn't find any implementation online so I'm writing my own) $\endgroup$ Feb 1, 2021 at 10:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .