0
$\begingroup$

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete state and action spaces.

In some examples of DQN use for solving Cart Pole gym environment, they use directly each state variable as input for the neural network (NN), therefore my doubt is: Can this input state variables for the NN be discretized in bins allowing to solve the cartpole environment ?

Thanks.

Improve this question
$\endgroup$
4
  • $\begingroup$ You could discretise the continuous output into bins and then use a tabular lookup table but you would either a) need to use very coarse discretisations, at which point you'd probably lose too much information to be able to solve the problem well b) require a very fine discretisation that would lead to a combinatorial blow up in number of states required to store in the lookup table. $\endgroup$
    – David
    Commented Apr 10 at 11:51
  • $\begingroup$ So what you're telling me is that is possible to discretize the state space and send that "bins" to the NN, though there're some problems of lossing information to effectively learn the goal ? $\endgroup$
    – Vitor Martins
    Commented Apr 10 at 13:15
  • $\begingroup$ oh, sorry, I didn't realise that you wanted to pass the discretised input into the NN. Why would you want to do this? I thought you wanted to learn in a tabular way. $\endgroup$
    – David
    Commented Apr 10 at 16:35
  • $\begingroup$ To use DQN algorithm. In a case that have to much discretized states and actions, i think that DQN should perform better than a Q-Learning algorithm. $\endgroup$
    – Vitor Martins
    Commented Apr 15 at 15:08

1 Answer 1

Reset to default
1
$\begingroup$

Binning of inputs are of course possible but it is a balancing act that may never outperform a network operating on continuous values.

Coarse binning may lose information necessary to accurately solve the problem. Imagine backup sensors on a car. If your goal is to get as close to the wall behind you without touching it, you can do it with a beeping sensor (distance binning sensor) but you won’t be as accurate as with a camera (continuous).

On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ok i understood, thanks. Relatively to this "On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.". Discretizing the state variables is not increasing the number of state variables is reducing the information of each variable. $\endgroup$
    – Vitor Martins
    Commented Apr 10 at 15:05
  • $\begingroup$ I wrote this very early in the morning for me. When I read descritizing my brain went to one hot encoding of the continuous input into bins which does tempt the curse. On the other hand, turning the continuous input into a step wise function (ie, round(x,0)) does not tempt the curse, you are correct. This nestles up to NN quantization and it's trade-offs. $\endgroup$
    – foreverska
    Commented Apr 10 at 15:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .