Reinforcement learning with $\gamma=0$ versus supervised framework [duplicate]

Question

I am on a learning phase (still to enter the details but I would like first to get a bird's eye overview). I would like to understand the difference between:

(1) a reinforcement learning framework, with $\gamma=0$, policy gradient optimization, and where the environment presents a new state at every iteration sampled from an input distribution. These conditions define a setup that we call RL0

(2) a standard neural network supervised classification framework. Call this second setup ANN.

To me the setups seem equal if we make the correspondence:

• ANN weights <-> RL0 policy parameters
• ANN inputs <-> RL0 states
• ANN predicted values <-> RL0 deterministic action given the state
• ANN loss function for one sample <-> RL0 reward given the state and action

My questions:

1. Is it really so? In other words can an ANN used in a supervised framework can be seen as a special case of reinforcement learning with a parametrized policy and discounted rewards with $\gamma=0$ ? Or am I missing some fundamental difference?

2. Does this mean that every RL technique applied in this setting should fall in some supervised learning category ? How would any RL technique applied to solve a supervised problem compare with a standard supervised framework?

NB: question Can supervised learning be recast as reinforcement learning problem? is very similar and answers point 1, but still left point 2 not very clear to me.

Answer 1

I add some thoughts of mine following Sutton and Barto, chapter 13, also to keep track of my work. This effort should go in the direction of answering point 2 of the question.

Policy gradient theorem

We consider the setting RL0 as in the OP, with the additional notation:

1. Each episod goes to the terminal state immediately after the first action.
2. Each state $s$ can be denoted as $x$, a member of the input state $X$, more suggestively remembering the notation in a supervised setting.
3. The action is the (deterministic) prediction of the parametrized model (e.g. a neural network), so that $\pi(a,s)$ can be rewritten as $\pi(y,x)=\delta(y-Y^{pred}(x,\theta))$, where $a$ has been substituted with $y$ to match notation with a supervised setting.
4. The reward is the squared error: $R=G=|Y^{pred}(x)-Y^{ex}(x)|^2$

Such a setting looks very similar to a supervised one. Indeed the policy gradient theorem reads:

$$\partial_{\theta} J=\int dy |Y^{pred}(x,\theta)-Y^{ex}(x)|^2\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y)) [1]$$

After some maninpulations and integration by parts one can rewrite the policy gradient theorem as above in the form:

$$\partial_{\theta} J=\partial_{\theta} |Y^{pred}(x,\theta)-Y^{ex}(x)|^2 [2]$$

which is the form one would use in a supervised form for the update. So up to the policy gradient theorem RL0 and ANN would be equal in this setting.

REINFORCE update

If we look at the REINFORCE update rule in this setting it reads:

$\delta \theta=|Y^{pred}(x,\theta)-Y^{ex}(x)|^2 \partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)$

To get a better understanding of the ratio of $\delta$ functions we rewrite:

$\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)=\delta'(Y^{pred}(x,\theta)-y)/\delta(Y^{pred}(x,\theta)-y)\partial_{\theta}Y^{pred}(x,\theta)$

So we would like to understand if the expression $\delta'(x)/\delta(x)$ can be given a finite meaning. Unluckily if we use a regularization of the delta funcion $\delta(x)=\frac{1}{\epsilon}e^{-\pi x^2/\epsilon^2}$ we see that this expression diverges when x is not null.

INTERPRETATION (for the moment):

It looks that the manipulations starting from the policy gradient theorem and leading to the reinforce update is where RL0 and ANN start to diverge substantially. I think the main issue there is that when in the derivation of the REINFORCE algorithm one multiplies and divides by $\pi(a|s)$, this operation provides larger estimators when $\pi(a|s)$ is low (because the $\pi$ at the denominator is the one surviving, the other gets replaces by sampling). As a consequence the REINFORCE update breaks when the densities are deterministic, i.e. when no sampling is necessary.

Comments welcome.