4
votes
Accepted
Action-value estimation of deterministic policies with Monte Carlo method
But, in a real application under a given deterministic policy $\pi$, how can you choose the initial action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?
...
- 23.1k
2
votes
Accepted
How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
So why is constant-$\alpha$ being used?
This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the ...
- 23.1k
2
votes
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
Your trajectories must contain rewards, so I'm assuming you've forgotten them in your original post, i.e., we must have $$\tau_j = (s_0^j, a_0^j, r_1^j, ..., s_{N_j}, a_{N_j}, r_{N_j+1})$$
Given that ...
2
votes
Accepted
When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?
You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$.
The $\geq$ inequality is also correct, of course, and it could be that the ...
- 1,763
1
vote
Accepted
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
What would be the standard approach in this case? Better use TD learning or Monte Carlo?
Both should be fine, but they might lead to different estimates, if both these things apply:
The amount of ...
- 23.1k
1
vote
Accepted
How to code an $\epsilon$-soft policy for on-policy Monte Carlo control?
You cannot code an $\epsilon$-soft policy directly, because it is not specific enough.
A policy is $\epsilon$-soft provided that there is at least a probability of $\frac{\epsilon}{|\mathcal{A}|}$ for ...
- 23.1k
1
vote
Accepted
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward?
By the return (sum of all rewards) from the whole episode. A lot of ...
- 23.1k
Only top scored, non community-wiki answers of a minimum length are eligible