2
$\begingroup$

I am on a learning phase (still to enter the details but I would like first to get a bird's eye overview). I would like to understand the difference between:

(1) a reinforcement learning framework, with $\gamma=0$, policy gradient optimization, and where the environment presents a new state at every iteration sampled from an input distribution. These conditions define a setup that we call RL0

(2) a standard neural network supervised classification framework. Call this second setup ANN.

To me the setups seem equal if we make the correspondence:

  • ANN weights <-> RL0 policy parameters
  • ANN inputs <-> RL0 states
  • ANN predicted values <-> RL0 deterministic action given the state
  • ANN loss function for one sample <-> RL0 reward given the state and action

My questions:

  1. Is it really so? In other words can an ANN used in a supervised framework can be seen as a special case of reinforcement learning with a parametrized policy and discounted rewards with $\gamma=0$ ? Or am I missing some fundamental difference?

  2. Does this mean that every RL technique applied in this setting should fall in some supervised learning category ? How would any RL technique applied to solve a supervised problem compare with a standard supervised framework?

NB: question Can supervised learning be recast as reinforcement learning problem? is very similar and answers point 1, but still left point 2 not very clear to me.

$\endgroup$
4
  • $\begingroup$ Hi. Yes the question is very similar indeed. I started also reading the article you linked. Indeed it looks that one can rewrite a supervised learning problem in a RL framework. $\endgroup$
    – Thomas
    Jun 9, 2023 at 14:01
  • $\begingroup$ So question 1 is solved. I have some follow up on question 2. You write in your answer to the other question "However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient)." $\endgroup$
    – Thomas
    Jun 9, 2023 at 14:04
  • $\begingroup$ This seems to suggest that the answer to question 2 is "no", because you (and also Barto in his chapter) suggest that using RL to answer a SL learning problem would be suboptimal. The follow up question would be than "what would be the differences in tackling a SL problem as a RL one and why this would be not an equivalent choice but a suboptimal one"? $\endgroup$
    – Thomas
    Jun 9, 2023 at 14:06
  • $\begingroup$ An effort to answer more quantitatively the second question is in the answer I wrote. Feedbacks welcome. $\endgroup$
    – Thomas
    Jun 16, 2023 at 7:52

1 Answer 1

0
$\begingroup$

I add some thoughts of mine following Sutton and Barto, chapter 13, also to keep track of my work. This effort should go in the direction of answering point 2 of the question.

Policy gradient theorem

We consider the setting RL0 as in the OP, with the additional notation:

  1. Each episod goes to the terminal state immediately after the first action.
  2. Each state $s$ can be denoted as $x$, a member of the input state $X$, more suggestively remembering the notation in a supervised setting.
  3. The action is the (deterministic) prediction of the parametrized model (e.g. a neural network), so that $\pi(a,s)$ can be rewritten as $\pi(y,x)=\delta(y-Y^{pred}(x,\theta))$, where $a$ has been substituted with $y$ to match notation with a supervised setting.
  4. The reward is the squared error: $R=G=|Y^{pred}(x)-Y^{ex}(x)|^2$

Such a setting looks very similar to a supervised one. Indeed the policy gradient theorem reads:

$$\partial_{\theta} J=\int dy |Y^{pred}(x,\theta)-Y^{ex}(x)|^2\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y)) [1]$$

After some maninpulations and integration by parts one can rewrite the policy gradient theorem as above in the form:

$$\partial_{\theta} J=\partial_{\theta} |Y^{pred}(x,\theta)-Y^{ex}(x)|^2 [2]$$

which is the form one would use in a supervised form for the update. So up to the policy gradient theorem RL0 and ANN would be equal in this setting.

REINFORCE update

If we look at the REINFORCE update rule in this setting it reads:

$\delta \theta=|Y^{pred}(x,\theta)-Y^{ex}(x)|^2 \partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)$

To get a better understanding of the ratio of $\delta$ functions we rewrite:

$\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)=\delta'(Y^{pred}(x,\theta)-y)/\delta(Y^{pred}(x,\theta)-y)\partial_{\theta}Y^{pred}(x,\theta)$

So we would like to understand if the expression $\delta'(x)/\delta(x)$ can be given a finite meaning. Unluckily if we use a regularization of the delta funcion $\delta(x)=\frac{1}{\epsilon}e^{-\pi x^2/\epsilon^2}$ we see that this expression diverges when x is not null.

INTERPRETATION (for the moment):

It looks that the manipulations starting from the policy gradient theorem and leading to the reinforce update is where RL0 and ANN start to diverge substantially. I think the main issue there is that when in the derivation of the REINFORCE algorithm one multiplies and divides by $\pi(a|s)$, this operation provides larger estimators when $\pi(a|s)$ is low (because the $\pi$ at the denominator is the one surviving, the other gets replaces by sampling). As a consequence the REINFORCE update breaks when the densities are deterministic, i.e. when no sampling is necessary.

Comments welcome.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged .