2
$\begingroup$

enter image description here

The above environment is DeepTraffic

Now consider this situation in the above environment, the Red car (we control it with our RL agent) is on the extreme right lane.

During the exploration phase, we take a 'move right' action, which ofcourse will result in the car not moving right,but the other cars will be moving, state changes due to the rules of the environment.

I'm using CNN to solve this, the state representation is the image itself and its a Q-Learning algorithm as described in DQN paper from deepmind.

In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?

and when remembering the state transition (s,a,r,s') should i remember the actual action 'move right'(invalid) or 'do nothing'(correct as per env) ?

$\endgroup$
  • 1
    $\begingroup$ This sounds like a great research question. $\endgroup$ – quintumnia Jan 18 '18 at 6:09
  • 2
    $\begingroup$ If the agent goes right as opposed to donny nothing, does it get a different reward? $\endgroup$ – Jaden Travnik Jan 18 '18 at 12:53
  • 1
    $\begingroup$ 'Do Nothing' action has 0 reward on all states. In this situation go-right will also have a 0 reward, since no damage is done. $\endgroup$ – Naveen Jan 18 '18 at 13:44
1
$\begingroup$

In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?

Generally the agent does not need to "think" what the next state will be. After taking an action, it should observe the reward and the next state.

The logic for the next state is controlled by the environment. In your case, the environment of the simulator should accept the action choice and then not allow the sideways movement off the road, and present the correct logical next state. It may in addition alter the reward signal if you want to penalise this choice (in comments you say this is not done - that's fine, it's a choice you make on problem set up, what behaviour to reward or penalise).

A more advanced agent might also try to predict what the next state will be, especially if it was using a planning algorithm (e.g. Dyna-Q). Even then there is no reason or rule that it should have to predict an impossible next state - a well-trained agent should in fact predict correctly that moving right is same as "do nothing" when the car is in far right-hand lane. For a value-based approach, like Q-learning the agent may do this simply by scoring the two actions with the same value.

In your case, you do not need to try and second-guess what the agent will predict. Provided your state information is enough for the agent to observe that it is in the far left or far right lane, it should [eventually, after training] correctly predict the value functions and/or optimal actions as it learns the environment. That includes, after a few examples, that it will predict the outcome when moving right when already on right hand side. For that to happen reliably, you do need to check your state representation is adequate, and function approximation hyper-parameters are suited to the problem (the latter part could take some experimentation)

If the state data that the car receives does not contain enough information for the agent to tell that it is in extreme left or right lane, then you will have a problem. This is entirely possible in general, although from the image you show, that does not seem to be an issue.

One possible related issue is if the other cars in the scene all have different velocities. This velocity information is not shown in the state, and could cause your agent problems as it cannot tell the difference between a stationary car in front of it, and one which better matches its own velocity. You could treat this missing data as a partially observable state, and adjust you agent. Or you might be able to do the same as the Deep Mind DQN team for Atari games and use a sequence of e.g. last 4 images as the state, which should then include enough velocity data to make reliable estimates of state/action values.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.