New answers tagged markov-decision-process
1
In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). However, the unichain assumption does not mean that every policy will eventually visit every state. I believe your confusion arises from the difference between ...
0
$\mathcal S$ is just a set of all possible states. It doesn't matter if it's agents perceived state or true environment state, they are within the same set of states. Agent cannot perceive itself to be in some "middle" state that's not in $\mathcal S$, it might think that's in the state that's not the actual environment state but that state is also ...
1
My understanding from your question is that you have the following data generated from a random policy:
$$[s_0, s_1, s_2 . . . s_n]$$
That is, the state observed at each time step.
You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do ...
2
The value function is defined as $v_\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ where $G_t$ are the (discounted) returns from time step $t$. The expectation is taken with respect to the policy $\pi$ and the transition dynamics of the MDP.
Now, as you pointed out the optimal value function is defined as $v_*(s) = \max_\pi v_\pi(s)\; ; \;\forall s \in \mathcal{S}$....
1
RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.
You need to provide details of your problem, if it is 1 step then any optimization system can be used.
0
The belief state in a POMDP is a distribution over the hidden state given all past actions and observations, i.e., at time $k$, the belief state is $b_k(s_k) \triangleq P(s_k \mid a_{0:k-1}, z_{1:k})$, where $a$ and $z$ are the actions and observations, respectively.
What you are asking about and calling "backtracking" boils down to the question: &...
1
The premise of this question is somewhat misleading. There is a deterministic optimal policy for a MDP, but this does not mean a stochastic optimal policy never exists. Talking about the optimal policy can be misleading, as there may be many different optimal policies.
For example, certainly we could imagine an MDP where $Q^*(s,a_0) = Q^*(s,a_1)$ for two ...
Top 50 recent answers are included
