In Monte Carlo-based action value estimation problem for a deterministic policy (estimation of $q_{\pi}(s,a)$),the estimation problem seems not to be well-defined because $q_{\pi}(s,a)$ by definition means the value of an arbitrary action $a$ at a given state $s$ when initial action $a$ is applied at that state and then following actions from policy $\pi$ at the next states. But, in a real application under a given deterministic policy $\pi$, how can you choose the initial action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
But, in a real application under a given deterministic policy $\pi$, how can you choose the initial action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?
You ignore $\pi$ for the action selection of the action you need to evaluate. This is well-defined, it is the expected future return for taking action $a$ in state $s$ and thereafter following the policy $\pi$.
So you may have a problem estimating that value, unless you also force that action choice during training in order to observe what transitions and rewards follow (that you would never see following the deterministic policy). However, it is well-defined conceptually.
A very simple way to force the assessment of all state/action pairs is exploring starts: Pick an arbitrary state/action pair to evaluate, then follow the transition and policy rules from that point on until the espidode end. This will give you a Monte Carlo sample of the value for the starting point, that you can use to update the estimate.
If you are not able to use exploring starts, or otherwise take actions different from the supplied deterministic policy, then you may be stuck. You would only have data of certain $s,a$ pairs. You could estimate $Q(s,a)$ for those pairs, and not for others.
answered Jan 29, 2022 at 21:36
-
$\begingroup$ @ Neil Slater In a real application, how can you pick an arbitrary state/action pair to evaluate? Maybe, the state/action pair you chose does not exist in episodes. $\endgroup$ Jan 29, 2022 at 22:03
-
$\begingroup$ The answer includes this - start your episode from the arbitrary state/action pair. That is called exploring starts. If you only have historical data, or are not able to set up the environment in particular states, then your options are limited. Probably you will not be able to complete the Q function, given your other constraint of a fixed deterministic policy. But maybe that's not important. You could still create a state value function for all reachable states for instance. $\endgroup$ Jan 29, 2022 at 22:09
-
$\begingroup$ @user3489173 You don't say in the question what your goal is. If you have a specific goal and constraints you must meet, I may be able to explain whether it is logically possible. $\endgroup$ Jan 29, 2022 at 22:12
-
$\begingroup$ @ Neil Slater. Assume that our application is a real-time application and the historical data of this application contains n number of episodes where n is a large number. I wonder how you can estimate action value function using monte carlo methods. $\endgroup$ Jan 29, 2022 at 22:19
-
$\begingroup$ @user3489173 If you have historical data using a fixed policy, you can only estimate action values of the actions that were taken. If the policy was deterministic, that will mean you know nothing about any alternative actions. You can fill in $Q(s,a)$ estimates for the actions that were observed, but not anything else. $\endgroup$ Jan 29, 2022 at 22:21