1
$\begingroup$

I am learning about deep learning. Currently just at a superficial level, I think I am misunderstanding how reinforcement learning and artificial neural networks are used together.

For what I first understood, artificial neural networks can be used as part of a broader RL solution, enhancing the so-called Q-learning algorithms that are normally utilised there. Yet after prompting ChatGPT, it seems that RL can also be used as a framework to train ANNs, but then it gets confusing (maybe it is hallucinating ?).

Improve this question
$\endgroup$
4
  • $\begingroup$ RL has nothing to do with ANN, though if you want to make it scalable, you use it with ANN (chatgpt is a ginormous compressor of a ngram model, thus you can replace with a huge table ideally) $\endgroup$
    – Alberto
    Commented Jun 30 at 1:04
  • $\begingroup$ But what about policy gradient methods like PPO that was introduced by OpenAI to fine tune their models ? (Just found that) $\endgroup$
    – Esteban
    Commented Jun 30 at 1:46
  • $\begingroup$ you can use PPO with tabular RL as well, so not sure what you mean $\endgroup$
    – Alberto
    Commented Jun 30 at 9:56
  • $\begingroup$ Your first answer suggests that we can use ANN to scale RL (for what I understand, when environment have large state spaces). So that's for "ANN used in RL". However, I also found after further research that we can apply the RL paradigm (find the best policy that maximise the cumulative reward if I am correct) to train ANN, whether by treating ANN as a policy (policy gradient method) or as an actor (actor-critic methods). It suggests that there is now two different relationships between ANN and RL to consider, but is it correct ? $\endgroup$
    – Esteban
    Commented Jun 30 at 14:35

3 Answers 3

Reset to default
1
$\begingroup$

RL is a learning algorithm, just like evolutionary strategies or SGD. ANNs can be used as the value function in RL, which is learned. The classic example is deep RL with AlphaGo. However, there are many RL methods which don’t use NNs, like Q learning as you mentioned.

A clarification by Sutton and Barto which may clear up the area of confusion: RL is both a general field of models/techniques as well as a set of learning algorithms. It depends on the context, but it seems like what you are referring to is RL as a method.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ RL is not a learning algorithm. It's a field/area of study. But there are RL algorithms. I think it's important to use the right terminology to avoid further confusion. $\endgroup$
    – nbro
    Commented Jul 4 at 10:56
2
$\begingroup$

So now that I inquired further about the question, here are some answers.

Generally speaking, when we use reinforcement learning (RL) and artificial neural networks (ANNs) in the same sentence, it's about using ANNs to approximate RL algorithms. In this case, we train the ANNs as we would normally do (Stochastic gradient descent...).

Here, the ANNs serve as the RL algorithms agents.

However, we can also use RL for :

  • Hyperparameter optimization and Neural architecture search (NAS), where ANNs are treated as the environment. We use RL algorithms to find the best architecture (number for layers...) or hyperparameters (activation function...). See Zoph, B., & Le, Q.V. (2017) "Neural Architecture Search with Reinforcement Learning" as an example.
  • Weights optimization (i.e. RL algorithms replacing the classical training algorithms), which is a very rare use case as our traditional optimizers prove to be already pretty efficient. See Bello, I., Zoph, B., Vasudevan, V., & Le, Q.V. (2017) "Neural Optimizer Search with Reinforcement Learning" as an example.

In those cases, the RL algorithms are the agent, and the ANN is the subject being optimised (environment).

Improve this answer
$\endgroup$
0
$\begingroup$

I would say that this is just be the semantics of the sentence you read. I would understand the sentence 'RL can be used as a framework to train an ANN' to mean that we are using an RL algorithm to learn the weights of an ANN -- an example would be to learn the parameters of a deep Q-network, which is used to approximate Q-values.

What's important to remember is that, as @Alberto points out in the comments, RL is generally separate to ANNs. RL is a branch of machine learning that can use ANNs as function approximators for the Q-function/policy (this is the context in which one might say that RL is 'training' the ANN), but is in general not tied to them (obviously you could argue that right now, they are tied to ANNs in the sense that to get them to work in real world settings, you need a deep network for the function approximations, but this might not always be true).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I fully understand the second part, the confusion remain in the first part, I'm trying to clarify : while you say it is possible to use an RL algorithm to learn-optimise the weights of an ANN, you talk about DQN, which I thought were precisely an example of ANN being used inside a RL solution, and that DQN are trained using the classic optimisers we know (like gradient descent etc...). I am rather expecting that if RL can be use to train ANN, it's by incorporating novel optimisers (same way genetic algorithm can be use to optimise ANN weights), but that might not be the case. $\endgroup$
    – Esteban
    Commented Jun 30 at 15:01
  • $\begingroup$ okay, yes, I can see based on the answer provided that this is the angle you were trying to understand. indeed, RL has been used in neural architecture search though it is quite limited currently. $\endgroup$
    – David
    Commented Jul 1 at 7:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .