1
$\begingroup$

I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.

Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?

Here are the two options I can think of.

  1. The input of the neural network is $(s_i, a_i)$ and the output is $Q(s_i,a_i)$

  2. The input is $(s_i)$ and the output is a vector $[Q(s_i,a_1), Q(s_i,a_2), \dots, Q(s_i,a_N)]$

Is there any other alternative architecture?

Also, how to reason about which model would be logically better?

$\endgroup$
1
  • $\begingroup$ Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the DQN paper. In any case, this question seems to be a duplicate of this one. $\endgroup$
    – nbro
    Dec 19, 2020 at 13:27

1 Answer 1

2
$\begingroup$

I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions. One main reason for this choice: when exploiting the learnt function (i.e. policy is $\arg \max_a Q(s,a)$), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .