I'm trying to find the optimal policy for the mountain car problem using deep Q learning with images as input, however, I cannot find a way to get my Q function to give me good solutions (I followed multiple tutorials for similar problems (Atari games and Flappy bird)). I'm working on Python with Keras.
The images are given in the following format :
400x400 pixels, where the bar on the bottom right corner represents the speed of the car.
To check where my problem might lie, I thought it would be best if I split the problem by first ensuring that I can find a network which would successfully find the state of the car (position and speed, since it's all we need to find it) by feeding my convolutional network images of random states (uniformly distributed within the state space).
After unsuccessful results, I decided to split it even more by only trying to find the two state variable separately.
This is the best kind of result that I get, when the network doesn't predict all the state to be the same (which seem to happen a lot with relu activation). The state space is divided in 50x50 matrix to make my predictions. The predicted speed is on the left, and the absolute error is on the right
The images fed to the network are pre-processed the follow way : 1. Gray scale 2. Resize (I tried 50x50, 100x100 and 150x150) 3. Values centered around 0 in [-1;1] (this seemed to help a bit with the relu activation)
The network I used to try to find the speed is first a convolution layer (I tried 32 windows of kernel_size=(4,4) (8,8) and (16,16), strides=(1,1) and (2,2), activation= relu, linear, tanh.
Optional additional convolution layers of kernel size half the previous layer.
And an optional last dense layer of dimension 32 or activation relu, linear or tanh.
The output layer is dimension one with linear activation.
The way I train the network is by feeding the fit function with 32 random samples and let the network train for 25 epochs with batch_size 32 and repeat ad libitum.
It's becoming extremely frustrating, especially with my GPU not fitting the requirement for GPU computation to check if I get some results faster.
Can anyone tell me if I'm doing something wrong that I'm missing and what can I do to improve my method to eventually manage to get the reinforcement algorithm to work ? Like the size of the training sample, batch size and epochs of the fit function, the structure of the network, ...
Edit : I finally found a way for my reinforcement learning to converge to the true Q value : each time I run a new episode and put it in my replay memory, I run many fits of different mini-batches. This is something I thought I did by increasing the number of epochs in the parameter of the fit function, but I guess it doesn't work as I thought it would. I'm letting it train a bit and then I will try the same method for the sub-problems mentioned above.