0
$\begingroup$

I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is designed to take some state and action and compute the value of that state-action pair.

Performing this process several times will eventually produce a table of states and what action will likely lead to a high reward.

In RL examples, I've seen them train a neural network to output q-values and a loss function like MSE to compute the loss between the q-learning update rule q-value and the NN's q value.

So:

(a) Q-learning update rule-> outputs target Q-values

(b) NN -> outputs Q values

MSE to compute the loss between (a) and (b)

So, given we already know what the target Q-value is from a, why do we need to train a NN?

| improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Can you please link us to the algorithm that you have seen where the target Q values are not produced by a NN but an NN is still used in the algorithm? $\endgroup$ – nbro♦ Jul 16 at 21:20
  • $\begingroup$ Hello, what I mean is the target is produced by a Q-learning update rule, and the NN is what is trained to mimic the behaviour of the Q-learning update rule. The example is from a book called "Deep RL in Action" and the example can be found on pages 54-68. $\endgroup$ – TNT Jul 16 at 22:22
3
$\begingroup$

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural nets come in handy when the state space is very large (or even continuous), so you can't afford to store a table of Q values. Also, neural nets have the ability to generalize across "similar" states -- for instance, if two states are very similar the neural net would likely produce similar values for those states, whereas a tabular implementation might not have seen enough data to accurately estimate the Q values of both.

| improve this answer | |
$\endgroup$
  • $\begingroup$ But surely for large state space problems, the same method would apply? There's an example in a book called "Deep Reinforcement Learning in Action" that has an example of a NN being trained to learn to play Gridworld, which is what I'm basing my question off. So I suppose what I'm asking is: do we use NNs because they're much more efficient at handling the task? $\endgroup$ – TNT Jul 16 at 17:58
  • 1
    $\begingroup$ In general, as stated above, nowadays, NNs are the preferred choice (over traditional, tabular approaches to RL) because of the reasons mentioned in the answer above. But if you apply NN to very simple problems, I see two main reasons: 1) It's super convenient to use them because super-efficient, scalable implementations are available for free on the internet; 2) often applying a NN-based RL-algorithm to a simple task can be seen as some kind of proof of concept (and for illustration). If you can show that it works nicely on small problems, then it will theoretically also work on larger ones. $\endgroup$ – Daniel B. Jul 16 at 18:37
  • 1
    $\begingroup$ Besides that, even for games that look super simple (superficially), the number of possible states can grow tremendously large pretty quickly. So, even if you could (still) fit all those states in a table, the required training time could be immense. Just because there is just no generalization in the training outcome when training a tabular RL agent. $\endgroup$ – Daniel B. Jul 16 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.