# Tag Info

Accepted

### Sutton & Barto: what are parametrized functions?

A parameterized function is a function that is defined by a set of parameters. If you change the parameters, you also change the actual function. For example, let's define this linear function f: \...
1 vote

### What is the difference between an on-policy distribution and state visitation frequency?

1.First of all. The on-policy distribution $\mu(s)$ is a probability distribution. So, obviously, it is different from state visitation frequency $\rho_\pi(s)$, since $\rho_\pi(s)$ is not normalized ...
1 vote

### $\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I dont understand why the $\gamma^t$ appears when you write the gradient with an expectation. Could you elaborate ? thank you I agree with you with all the things up to that point EDIT : to try to ...
1 vote
Accepted

### Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?

To expand $\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s]$, you can take the same expectation over next state and reward as for $R_{t+1}$ (in fact this is normally shown without separating the two terms as you ...
1 vote

### Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?

Based upon your reference background of the given sample episodes in one batch, you need to average v_pr(2) since there're 8 cases then update your vector v with vector v_pr before next batch before ...
1 vote
Accepted

### How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

Short explanation The bias converges asymptotically to zero with more visits of the state $s$. The value function is estimated in the following way: \begin{equation} v_{\pi}(s) = \frac{\sum_{t \in \...
1 vote

### Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

The answer is: $\sum_{s} \eta(s)$ is not a constant with regards to $\theta$. As you already mentioned \begin{equation} \sum_{k=0}^{\infty} \text{Pr}(s_{0}\rightarrow s, k, \pi) = \eta(s). \end{...

Only top scored, non community-wiki answers of a minimum length are eligible