# Tag Info

7

Initial state How things are at first. In your particular example, it would be where your k knights are placed on the board initially. Your problem doesn't precisely state this, so you could either place them at the bottom or at random. Goal state The board with the k knights placed on the target squares. State transition function A function that takes ...

3

Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance,...

3

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we condition on taking action $a$ then the expected reward at time $t+1$ is given as follows: \begin{align} \mathbb{E}[R_{t+1} | S_t = s, A_t=a] & = \sum_{s',...

1

I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but ...

1

I assume you use the 12 discrete features as state variables, and, for each of these variables, you will have at least two values. So, the minimum number of states will be $2^{12} = 4096$, which gives $(2^{12})^{2} = 16777216$ possible transitions. In order to reach this, you will need a huge amount of simulations, also taking into account that this number ...

1

In non-trivial cases, the transition matrix is (generally) not maintained in the traditional tabular form. If the representation used factored notation (Factored MDP) then Dynamic Bayesian networks can be used. Another approach would be to abstract state spaces so that you have reduced number of states in the representation of P. These can be roughly ...

Only top voted, non community-wiki answers of a minimum length are eligible