# How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to test this algorithm assumes R as a function of only states.

My question is:

Given $$\mathcal{X}$$ as the set of states of an MDP and $$\mathcal{A}$$ as the set of actions of an MDP. Supposing I have four states ($$1$$,$$2$$,$$3$$,$$4$$), two actions $$a$$ and $$b$$ and a reward function $$R: \mathcal{X}\to\mathbb{R}$$ s.t.

$$R(1) = 0$$

$$R(2) = 0$$

$$R(3) = 0$$

$$R(4) = 1$$

If I need to change the current reward function to a new reward function $$R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$$ is it ok to compute it as $$\forall a,R(s,a) = R(s)$$?

$$R(1,a) = 0$$

$$R(1,b) = 0$$

$$R(2,a) = 0$$

$$R(2,b) = 0$$

$$R(3,a) = 0$$

$$R(3,b) = 0$$

$$R(4,a) = 1$$

$$R(4,b) = 1$$

More generally, what's the correct way of generalising a reward function $$R: \mathcal{X}\to\mathbb{R}$$ to a reward function $$R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$$?

As explained here, I can write $$R(s,a) = R(s)\ \forall a$$ since the reward of my specific MDP is dependent exclusively to the state $$s$$.
• Just a clarification, if you're trying to reproduce a certain paper that uses $R(s, a)$, why do you say that in your MDP the reward function only depends on $s$? I suppose you're testing their approach in a simpler environment. Right? Is the code you're using available? – nbro Jan 31 at 19:04