0
$\begingroup$

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.

Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.

$\endgroup$
3
  • $\begingroup$ Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this? $\endgroup$
    – nbro
    Commented Jul 5, 2020 at 23:40
  • $\begingroup$ Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that). $\endgroup$ Commented Jul 5, 2020 at 23:58
  • $\begingroup$ You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question). $\endgroup$
    – nbro
    Commented Jul 6, 2020 at 0:04

1 Answer 1

1
$\begingroup$

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.

With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.

$\endgroup$
1
  • $\begingroup$ Ah, you're right, my mistake. Edited my response to address this. $\endgroup$
    – harwiltz
    Commented Jul 6, 2020 at 15:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .