3
$\begingroup$

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ is a fixed state I get to after taking the action $a$. Note that we can have situations where in multiple iterations we have: $$(s,a) \to (s_1, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_2) \\ (s,a) \to (s_3, r_3) \\ \vdots$$

My question is, would $r_1 =r_2$?

$\endgroup$
2
$\begingroup$

My question is, would $r_1 =r_2$?

That's usually up to you as the designer of the system.

Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are fixed values depending on $(s,a)$. So in your examples, you would expect your observations to also have $r_1 = r_2$

However, it is possible to define a MDP where transition to state $s'$ is deterministic, but $r$ is not. For instance, you could define reward in a game equal to the sum of a number of dice rolled, with better rewards (on average) resulting in more dice. This is still a valid MDP and can be solved using RL techniques.

A real-world example of this might be managing a queue of work, where you want to minimise lead time, but don't know for certain how long each task will take. Your state progression moves deterministically - you have a queue of pending tasks, current tasks and workers, and assigning a task to a worker is completely deterministic. However, you don't know how efficiently tasks will be performed until after they are done, so you don't know the reward perfectly from the assignment (whether you can treat this as random or hidden state is a more complex issue - it is often pragmatic to treat such unknown data as random though).

$\endgroup$
-1
$\begingroup$

I think being deterministic usually means that $s'$ comes from some probability distribution based on $s$ and $a$. In this case I think the rewards could vary based n the design.

$\endgroup$
  • 2
    $\begingroup$ Deterministic means that $s'$ can only take a single value. If it is drawn from some probability distribution where it can take more than one value, then the environment is stochastic and not deterministic $\endgroup$ – Neil Slater Feb 12 at 12:02
-2
$\begingroup$

The other answers seem to have different ideas of what a model-based setup means. It usually means that $p(r,s'|s,a)$ for all $r,s',s,a$ is known. I believe in this formulation, if we know this probability function in advance, then this is what "deterministic" means. Again there could be a one to many relationship between $s'$ and $r$.

$\endgroup$
  • 2
    $\begingroup$ "I believe in this formulation, if we know this probability function in advance, then this is what "deterministic" means". This is incorrect. Deterministic means that $𝑝(𝑟,𝑠′|𝑠,𝑎)$ must strictly be 0 or 1 with no in-between values. I.e. if you know the model, and know $s,a$ then you automatically know $s', r$ and you do not need to express that knowledge as a probability distribution (you could do so, but it would be trivial) $\endgroup$ – Neil Slater Feb 12 at 16:50

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.