New answers tagged

2 votes
Accepted

$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

Question: Can I write it without the subscript? So $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$ Yes, your reasoning is sound, there is no need to condition the expectation on the policy, ...
user avatar
  • 23.8k
4 votes
Accepted

In Value Iteration, why can we initialize the value function arbitrarily?

If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
user avatar
  • 928
6 votes

In Value Iteration, why can we initialize the value function arbitrarily?

Is this something to do with the Bellman optimality constraint itself? That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
user avatar
  • 23.8k
1 vote
Accepted

Why does the average-reward estimator for continuing tasks use the TD error?

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward. With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$...
user avatar
  • 66

Top 50 recent answers are included