0
$\begingroup$

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.

Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.

$\endgroup$
3
  • $\begingroup$ Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this? $\endgroup$
    – nbro
    Jul 5 '20 at 23:40
  • $\begingroup$ Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that). $\endgroup$ Jul 5 '20 at 23:58
  • $\begingroup$ You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question). $\endgroup$
    – nbro
    Jul 6 '20 at 0:04
1
$\begingroup$

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.

With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.

$\endgroup$
1
  • $\begingroup$ Ah, you're right, my mistake. Edited my response to address this. $\endgroup$
    – harwiltz
    Jul 6 '20 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.