7

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function. The important part of the image is when Model-Free RL splits into Policy Optimization (which ...


2

Is it possible for value-based methods to learn stochastic policies? Yes, but only in a limited sense, due to the ways it is possible to generate stochastic policies from a value function. For instance, the simplest exploratory policy used by SARSA and Monte Carlo Control, $\epsilon$-greedy, is stochastic. SARSA natually learns the optimal $\epsilon$-...


2

Assuming a continuous/uncountable state space, we can only estimate our value function using function approximation, so our estimates will never be true for all states simultaneously (because, loosely speaking, we have far more states than weights). If we can look at the (approximated) value of states we take in, say, 5 actions time, it is better to make a ...


1

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$. However, you can recover $Q^{\pi}(s,a)$ values ...


Only top voted, non community-wiki answers of a minimum length are eligible