# Tag Info

Accepted

### What are the state space and the state transition function in AI?

Initial state How things are at first. In your particular example, it would be where your k knights are placed on the board initially. Your problem doesn't ...
• 575
Accepted

### If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we ...
• 4,920

### Is it appropriate to represent 'total failure' as an absorbing state?

In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For ...
• 32.7k
Accepted

### How can we find the value function by solving a system of linear equations without knowing the policy?

Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through ...
• 32.7k

### How do compute the table for $p(s',r|s,a)$ (exercise 3.5 in Sutton & Barto's book)?

The function $r(s,a,s')$ gives the expected reward in each scenario, but not the distribution of rewards that lead to values $r_{search}$ and $r_{wait}$ The text explains that reward is $+1$ for ...
• 32.7k

### How do compute the table for $p(s',r|s,a)$ (exercise 3.5 in Sutton & Barto's book)?

At first, like Neil Slater says, I thought this could only be solved using the expected rewards instead of actual rewards, or else there wasn't enough information to solve it. But now I think there ...
• 181
1 vote

### Can the state transition function be dynamic in reinforcement learning?

Is it possible to make the transition function change as the game progress? Yes, the normal way to do this would be to make the current time or time step $t$ part of the state $s$, so that equation 1 ...
• 32.7k
1 vote

### What is the difference between a distribution model and a sampling model in Reinforcement Learning?

I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily model something, unless, for example, you are ...
• 40.9k
1 vote

### How do compute the table for $p(s',r|s,a)$ (exercise 3.5 in Sutton & Barto's book)?

This means that the reward set is actually R={0,1,−3} (we assume that in each timestep, the robot can only collect one can). @riceissa While I agree with the rest of your demonstration, I wouldn't ...
1 vote

### How do compute the table for $p(s',r|s,a)$ (exercise 3.5 in Sutton & Barto's book)?

In the announced problem, most of the transitions aren't possible, so most the terms of equations (3.3) and (3.4) from the book will end up being 0. In my understanding,  \begin{align} p(s'= high |...
• 111
1 vote
Accepted

### How to fill in missing transitions when sampling an MDP transition table?

I assume you use the 12 discrete features as state variables, and, for each of these variables, you will have at least two values. So, the minimum number of states will be $2^{12} = 4096$, which gives ...
• 375
1 vote

### How do you generate the transition probabilities of a non-trivial MDP?

In non-trivial cases, the transition matrix is (generally) not maintained in the traditional tabular form. If the representation used factored notation (Factored MDP) then Dynamic Bayesian networks ...
• 151

Only top scored, non community-wiki answers of a minimum length are eligible