# How to estimate a behavior policy for off-policy learning based on data?

I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $$p(r,s' \mid s,a)$$.

Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?

Is there a way to estimate a behavior policy from this dataset so that it can be used in an off-policy learning algorithm?

If you have enough examples of $$(s,a)$$ pairs for each instance of $$s$$ then you can simply estimate

$$b(a|s) = \frac{N(a,s)}{N(s)}$$

Where $$N$$ counts the number of instances in your dataset. This might be enough to use off-policy with importance sampling.

Alternatively, you can use an off-policy approach that doesn't need importance sampling. The most straightforward one here would be single-step Q learning. The update step for 1-step Q-learning does not depend on behaviour policy, because:

• The action value being updated $$Q(s,a)$$ already assumes $$a$$ is being taken, so you don't need any conditional probability there.

• The TD target $$r + \gamma \text{max}_{a'}[Q(s',a')]$$ does not need to be adjusted for behaviour policy, it works with the target policy directly (implied as $$\pi(s) = \text{argmax}_{a}[Q(s,a)]$$)

A 2-step Q learning algorithm would need to adjust for likelihood $$b(a'|s')$$ in the TD target $$\frac{\pi(a'|s')}{b(a'|s')}(r + \gamma r' + \gamma^2\text{max}_{a''}[Q(s'',a'')])$$ - typically $$\pi(a'|s')$$ is either 0 or 1, thus making $$b(a'|s')$$ irrelevant some of the time. But you would still prefer to know it for performing updates it you can.

If you are making updates offline and off-policy, then single-step Q learning is probably the simplest approach. It will require more update steps overall to reach convergence, but each one will be simpler.

You can simply train a policy from the inputs to predict the actions in your dataset. You can use the cross entropy loss for this, i.e. maximize the the log probability that the policy assigns to the actions in the data set when given the corresponding inputs. This is called behavioral cloning.

The result is an approximation of the behavioral policy that lets you compute probability densities of actions. It is an approximation because the dataset is finite, and even more so when you restrict the learned policy to a class of distributions, e.g. Gaussians.

If your data look like this $$(s_{1},a_{1},r_{1},s_{2}),(s_{2},a_{2},r_{2},s_{3}),....,$$ then this sample drawn from a particular behavior policy. So, you do not need to find the behavior policy just Q-Learning to find the optimal policy while following the behavior policy.

If the MDP is too big then consider applying Deep Q Learning. In both cases, the transition probability they have given has no use. But if you use on-policy learning and you know the dynamics of the system(means transition probabilities), I will recommend you to use dynamic programming(if state-space is not quite large). But for your above problem setting, you can not use dynamic programming, you have only one choice to use off-policy learning.

• The behavior policy is given, so there is no need to find the behavior policy(since this data is drawn from a particular behavior policy). – Swakshar Deb Apr 27 '20 at 15:30
• The samples you have drawn is from the behavior policy. Someone gave him access to this data. So now he can just use Q-Learning to find the optimal policy. In case of Q-Learning, you do not need the $b(a|s)$. – Swakshar Deb Apr 27 '20 at 15:34
• The edit makes it better. Might be worth clarifying that you mean single-step Q-learning, although that is the most common variant. – Neil Slater Apr 27 '20 at 15:38
• Yes, this statement "So, you do not need to find the behavior policy" is wrong in some cases. But for Q-Learning you still do not need to know this behavior policy. – Swakshar Deb Apr 27 '20 at 15:40
• I'm going backwards and forwards on n-step Q learning :-). Actually I stand by my first statement - you need to know behaviour policy for n-step Q learning, because the wieghtings may vary when considering any non-zero update. – Neil Slater Apr 27 '20 at 15:55