New answers tagged

2 votes
Accepted

Is the sequence 1-1-2-3-Exit possible in the following Markov reward process?

No, it's not possible, and you correctly explained why.
nbro's user avatar
  • 40.3k
2 votes

If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?

You've correctly figured out the most part of your confusion. The only thing I want to add is that by the finite MDP definition here it's certainly possible $p(s'|s,a)=0$ for some $s',s$. The reason ...
cinch's user avatar
  • 1,381
1 vote
Accepted

If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?

Ok, so I was able to find the answer to this question by myself. So I'm sharing it with everyone. By the law of non-exclusive events, $P(A|B) = \frac{P(A,B)}{P(B)}$ where $P(B) > 0$. This can also ...
Jahid Chowdhury Choton's user avatar
1 vote
Accepted

What is the logic in including/not including subscript $\pi$ in in "E" for value functions?

Your second equation is defined on page 78 and in the same page there's a step on the lower part of derivations contains the answer to your confusion. $q_{\pi}(s, {\pi'}(s)) \\= E[R_{t+1} + \gamma v_{...
cinch's user avatar
  • 1,381
0 votes

$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

@Neil Slater In the expected answer it is said that "This only works because you have split off the expected reward to evaluate separately. Overall, the value of $q_{\pi}$ does depend on the ...
DSPinfinity's user avatar
1 vote

What is the logic in including/not including subscript $\pi$ in in "E" for value functions?

The subscript just indicates over which variable the expectation is taken. For consistency it can be included everywhere. Sometimes, when the variable over which the expectation is taken is obvious, ...
vl_knd's user avatar
  • 476
2 votes
Accepted

Is there any example of a Markov Decision Process (MDP) with infinite number of states?

Many real-world problems have an infinite number of states, but, in practice, digital computers cannot represent an infinite number of states or numbers anyway, so you will always need to discretize ...
nbro's user avatar
  • 40.3k
1 vote
Accepted

When is it non-Markovian?

Your observation about the possibility of n being infinite makes sense. If an environment requires an infinite unbounded history of states to make decisions, it is considered non-Markovian. However, ...
cinch's user avatar
  • 1,381

Top 50 recent answers are included