In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology.
Suppose my state is a high dimensional vector of length $N$, where $N$ is a huge number. Let's say I solve this task using $Q$-learning and I fix the state space to $10$ vectors, each of $N$ dimensions. $Q$-learning can easily work with these settings as we need only a table of dimensions $10$ x number of actions.
Let's say my state space can have an infinite number of vectors each of $N$ dimensions. In these settings, Q-learning would fail as we cannot store Q-values in a table for each of these infinite vectors. On the other hand, DQN would easily work, as neural networks can generalize for other vectors in the state-space.
Let's also say I have a state space of infinite vectors, but each vector is now of length $2$, i.e., small dimensional vectors. Would it make sense to use DQN in these settings? Should this state-space be called high dimensional or low dimensional?