# How can reinforcement learning be unsupervised learning if it uses deep learning?

I was watching a video in my online course where I'm learning about A.I. I am a very beginner in it.

At one point in the course, the instructor says that reinforcement learning (RL) needs a deep learning model (NN) to perform an action. But for that, we need expected results in our model for the NN to learn how to predict the Q-values.

Nevertheless, at the beginning of the course, they said to me that RL is an unsupervised learning approach because the agent performs the action, receives the response from the environment, and finally takes the more likely action, that is, with the highest Q value.

But if I'm using deep learning in RL, for me, RL looks like a supervised learning approach. I'm a little confused about these things, could someone give me clarifications about them?

### Supervised learning

The supervised learning (SL) problem is formulated as follows.

You are given a dataset $$\mathcal{D} = \{(x_i, y_i)_{i=1}^N$$, which is assumed to be drawn i.i.d. from an unknown joint probability distribution $$p(x, y)$$, where $$x_i$$ represents the $$i$$th input and $$y_i$$ is the corresponding label. You choose a loss function $$\mathcal{L}: V \times U \rightarrow \mathbb{R}$$. Then your goal is to minimize the so-called empirical risk

$$R_{\mathcal{D}}[f]=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(x_i, f(x_i)) \tag{0}\label{0}$$

with respect to $$f$$. In other words, you want to find the $$f$$ that minimizes the average above, which can also be formally written as $$f^* = \operatorname{argmin}_f R[f] \tag{1}\label{1}$$ The problem \ref{1} is called the empirical risk minimization because it is a proxy problem for the expected risk minimization (but you can ignore this for now).

### Reinforcement learning

In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step $$t$$, the agent takes $$a_t$$ in the state $$s_t$$, receives a reward $$r_t$$ from the environment and the agent and the environment move to another state $$s_{t+1}$$.

The goal of the agent is to maximize the expected return

$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$

where $$t$$ is the current time step (so we don't care about the past), $$R_i$$ is a random variable that represents the probable reward at time step $$i$$, and $$G_t = \sum_{i=t+1}^\infty R_i$$ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $$t$$), which is also a random variable.

In this context, the most important job of the programmer is to define a function $$\mathcal{R}(s, a)$$, the reward function, which provides the reinforcement (or reward) signal to the RL agent. $$\mathcal{R}(s, a)$$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $$a$$ in the state $$s$$. (Note that $$\mathcal{R}$$ is different from $$R_i$$, which is a random variable that represents the reward at time step $$i$$).

### What is the difference between SL and RL?

In RL, you (the programmer) need to define the reward function $$\mathcal{R}$$ and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset $$\mathcal{D}$$, you choose $$\mathcal{L}$$ in \ref{0}, and the goal is to find the function $$f^*$$ that minimizes the empirical risk. So, these have different settings and goals, so they are different!

However, every SL problem can be cast as an RL problem. See this answer. Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.

### Is RL an unsupervised learning approach?

In RL, you do not tell the agent what action it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input $$x_i$$, the output should be $$y_i$$.

Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.

### What is deep learning?

The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.

In Supervised learning, the goal is to learn a mapping from points in a feature space to labels. So that for any new input data point, we are able to predict its label. whereas in Unsupervised learning data set is composed only of points in a feature space, i.e. there are no labels & here the goal is to learn some inner structure or organization in the feature space itself.

Reinforcement Learning is basically concerned with learning a policy in a sequential decision problem. There are some components in RL that are “unsupervised” and some that are “supervised”, but it is not a combination of “unsupervised learning” and “supervised learning”, since those are terms used for very particular settings, and typically not used at all for sequential decision problems.

In Reinforcement Learning, we have something called as Reward function that the agent aims to maximize. During the learning process, one typical intermediate step is to learn to predict the reward obtained for a specific policy.

In a nutshell, we can say Reinforcement Learning put a model in an environment where it learns everything on its own from data collection to model evaluation. It is about taking suitable action to maximize reward in a particular situation. There is no answer but the reinforcement agent decides what to do to perform the given task. In the absence of training dataset, it is bound to learn from its experience.

To better understand let’s look at an analogy.

Suppose you have a dog that is not so well trained, every time the dog messes up the living room you reduce the amount of tasty foods you give it (punishment) and every time it behaves well you double the tasty snacks (reward). What will the dog eventually learn? Well, that messing up the living room is bad.

This simple concept is powerful. The dog is the agent, the living room the environment, you are the source of the reward signal (tasty snacks).

To learn more on Reinforcement learning, pls check this awesome Reinforcement Learning lecture that is freely available on youtube by someone who actually leads the Reinforcement learning research group at DeepMind and also a lead researcher on AlphaGo, AlphaZero.

[RL Course by David Silver] https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB"