2
$\begingroup$

I'm trying to find the optimal policy for the mountain car problem using deep Q learning with images as input, however, I cannot find a way to get my Q function to give me good solutions (I followed multiple tutorials for similar problems (Atari games and Flappy bird)). I'm working on Python with Keras.

The images are given in the following format :

input

400x400 pixels, where the bar on the bottom right corner represents the speed of the car.

To check where my problem might lie, I thought it would be best if I split the problem by first ensuring that I can find a network which would successfully find the state of the car (position and speed, since it's all we need to find it) by feeding my convolutional network images of random states (uniformly distributed within the state space).

After unsuccessful results, I decided to split it even more by only trying to find the two state variable separately.

This is the best kind of result that I get, when the network doesn't predict all the state to be the same (which seem to happen a lot with relu activation). The state space is divided in 50x50 matrix to make my predictions. The predicted speed is on the left, and the absolute error is on the right

state_error.

The images fed to the network are pre-processed the follow way : 1. Gray scale 2. Resize (I tried 50x50, 100x100 and 150x150) 3. Values centered around 0 in [-1;1] (this seemed to help a bit with the relu activation)

The network I used to try to find the speed is first a convolution layer (I tried 32 windows of kernel_size=(4,4) (8,8) and (16,16), strides=(1,1) and (2,2), activation= relu, linear, tanh.

Optional additional convolution layers of kernel size half the previous layer.

And an optional last dense layer of dimension 32 or activation relu, linear or tanh.

The output layer is dimension one with linear activation.

The way I train the network is by feeding the fit function with 32 random samples and let the network train for 25 epochs with batch_size 32 and repeat ad libitum.

It's becoming extremely frustrating, especially with my GPU not fitting the requirement for GPU computation to check if I get some results faster.

Can anyone tell me if I'm doing something wrong that I'm missing and what can I do to improve my method to eventually manage to get the reinforcement algorithm to work ? Like the size of the training sample, batch size and epochs of the fit function, the structure of the network, ...

Edit : I finally found a way for my reinforcement learning to converge to the true Q value : each time I run a new episode and put it in my replay memory, I run many fits of different mini-batches. This is something I thought I did by increasing the number of epochs in the parameter of the fit function, but I guess it doesn't work as I thought it would. I'm letting it train a bit and then I will try the same method for the sub-problems mentioned above.

$\endgroup$
  • $\begingroup$ Welcome to AI! (I think there is a rendering error in the second set of graphics--I'm seeing the numbers overlap, resulting in illegibility.) $\endgroup$ – DukeZhou May 12 '18 at 22:59
  • $\begingroup$ Can you please provide a figure of the resized image? Are you using a sliding window to find the car? $\endgroup$ – Charles May 13 '18 at 1:13
  • $\begingroup$ The solution in your edit is "re-work the experience replay logic", of which there was no hint that this was the problem before your edit . . . Your question is interesting. However, your edit is revealing that the solution is in the details of the project - hyperparameters and implementation details. It is unlikely anyone here can take even the quite long description here and analyse the fault which is essentially "my learning algorithm is performing badly". At best you can expect well-meaning answers that suggest something to check that you may or may not already be doing. $\endgroup$ – Neil Slater May 13 '18 at 8:57
  • $\begingroup$ "there was no hint that this was the problem" If I knew it was the problem, I wouldn't have looked for help, would I ?;) Basically the reason why my protocol didn't work(or would take ages to converge (maybe because I use CPU ?)) is because I trained only one mini batch at a time in-between each episodes, which is what every algorithm I've looked at seemed to be doing (and why I didn't think about mentioning it). The change I made for it to work (in a reasonable amount of time) is to train over many random mini batches in-between episodes, since creating the image is time consuming. $\endgroup$ – user3548298 May 13 '18 at 12:57
1
$\begingroup$

The policy for solving the Mountain car problem is very complex. As far as i know from a human demonstration the solution is, that the car must first drive a bit backwards and then forward. To guide the neural network into that direction it make sense to define important keyframes first. The idea is to train the network for reaching only subgoals. And then combine the subgoals into a sequence of actions.

To match this idea to the concrete problem the neural network has to be trained to move the car a bit backwards. That means to reduce the error between the current position of the car and the carposition in the next keyframe. Finding such a policy is a lot easier then solving the overall problem.

enter image description here

Perhaps a small explanation about the figure. There are 3 keyframes visible: the starting position which is equal to the situation in the OP, the subgoal and the goal. Under the keyframes a RDF-like syntax for describing the task is given. This is used as input parameter for training the neural network. The overall picture is described in the literature as annotated keyframes, sometimes the concept is called “semantic event chains” which means basically to give a high-level task model for the domain to support the solver.

$\endgroup$
  • 1
    $\begingroup$ I'm familiar with the problem and had no trouble finding the optimal solution using the state (position and speed of the car) with a regression tree and an epsilon greedy policy. The problem I am facing here and what I don't understand is that I cannot even get a good approximation of the state from the images. $\endgroup$ – user3548298 May 12 '18 at 17:09
  • $\begingroup$ Also I have to assume that I don't know anything about what the optimal policy would look like (going backward). I can only use the classical reward (+1 if I reach the flag, -1 if I go out of the domain, 0 otherwise) $\endgroup$ – user3548298 May 12 '18 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.