Methods of constructing input and ouput vectors in Reinforcement Learning with approximation function learning?

Question

Give the one-hot state vector $\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$ and action spaces $A(s)=\{a_1,a_2\}$ for all $s$.

In a course, I was taught to construct "stack" input vectors like $[x_{11}(s,a),x_{21}(s,a),x_{12}(s,a),x_{22}(s,a)]^T$ with $x_{ij}(s)=1$ if the action is $a_j$ and the state is $s_i$, and the output is just $q(s,a)$.

However, in DQN, the input vector is $\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$, and the output vector is $[q(s,a_1),q(s,a_2)]$.

My question is: What are the usual ways to construct input and ouput vectors in Reinforcement Learning to learn the action value function q(s,a)? Are there any other methods?

Answer 1

If you build a function like $Q(s,a)$ using DQN, you have the problem that given 100 actions, you'll need 100 forward pass of your network

Now, since neural networks can handle multiple outputs, we usually (not always, for example not for continuous action spaces) just model $Q(s)$, where the network outputs all the Qs for all the actions. This way, you only need 1 forward pass to get all the Qs.

Also, bare in mind that using a 1-hot encoding, even though they make perfect sense for tabular cases, as state representation might not be the best for neural networks, as it will be much harder for them to generalize, and you'll have a giant input space