6
votes
Accepted
Sutton & Barto: what are parametrized functions?
A parameterized function is a function that is defined by a set of parameters. If you change the parameters, you also change the actual function. For example, let's define this linear function
$$f: \...
- 39.5k
1
vote
What is the difference between an on-policy distribution and state visitation frequency?
1.First of all. The on-policy distribution $\mu(s)$ is a probability distribution. So, obviously, it is different from state visitation frequency $\rho_\pi(s)$, since $\rho_\pi(s)$ is not normalized ...
- 2,426
1
vote
$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)
I dont understand why the $\gamma^t$ appears when you write the gradient with an expectation. Could you elaborate ? thank you
I agree with you with all the things up to that point
EDIT : to try to ...
- 11
1
vote
Accepted
Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
To expand $\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s]$, you can take the same expectation over next state and reward as for $R_{t+1}$ (in fact this is normally shown without separating the two terms as you ...
- 30.2k
1
vote
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Based upon your reference background of the given sample episodes in one batch, you need to average v_pr(2) since there're 8 cases then update your vector v with vector v_pr before next batch before ...
- 940
1
vote
Accepted
How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)
Short explanation
The bias converges asymptotically to zero with more visits of the state $s$. The value function is estimated in the following way:
\begin{equation}
v_{\pi}(s) = \frac{\sum_{t \in \...
- 322
1
vote
Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?
The answer is: $\sum_{s} \eta(s)$ is not a constant with regards to $\theta$. As you already mentioned
\begin{equation}
\sum_{k=0}^{\infty} \text{Pr}(s_{0}\rightarrow s, k, \pi) = \eta(s).
\end{...
- 322
Only top scored, non community-wiki answers of a minimum length are eligible