0
$\begingroup$

I am trying to use images to predict the sensor data of a racing game.

Being a bit of a newcomer I have multiple questions. All help/suggestion is appreciated.

Dataset

The dataset looks something like:

  • input - image: 160x120 grayscale from the bumper view - here is an example of some images here is an example of some images
  • output - sensors: vector of 21 elements, all normalized between [0, 1], representing the 3 sensors. Those sensors are:
    • angle between the car and the track axis (sensors[0])
    • 19 rangefinders, returning the distance from the car to the track limit, spanning from -pi and pi (sensors[1:20])
    • distance from track axis (sensors[20])

Here is an example of a sensor vector.

[
0.01011692 # angle
0.059058   0.299319   0.23943199 0.20102449 0.18029851
0.1706595  0.165723   0.161521   0.15858699 0.15570949 0.15288849
0.150124   0.146348   0.142166   0.1347065  0.121228   0.102669
0.08340649 0.04948675 # rangefinders
0.00183716 # distance from center
] 

I generated about 50000 entries in the dataset. In case this is not enough, increasing the size of the dataset is trivial as its creation is an entirely automated process.

Reasons and goal

The end goal is to use the sensors predicted from game frames to drive the car in real time. This way the car can be driven using only images.

The quality of the estimation from both models is not good enough, as the "driver" beheaves weirdly or has no idea of what to do.

Progress and results

I started with a plain CNN:

    model = Sequential()

    model.add(Conv2D(8, (4, 4), input_shape = (img_height, img_width, stack_depth), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(8, (4, 4), padding="same", strides = 2, activation = "relu"))
    model.add(Conv2D(8, (4, 4), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(8, (4, 4), padding="same", strides = 2, activation = "relu"))
    model.add(Conv2D(16, (3, 3), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(16, (3, 3), padding="same", strides = 2, activation = "relu"))
    model.add(Conv2D(16, (3, 3), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(16, (3, 3), padding="same", strides = 2, activation = "relu"))
    model.add(Conv2D(32, (3, 3), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(32, (3, 3), padding="same", strides = 2, activation = "relu"))
    model.add(Conv2D(32, (3, 3), padding="same", activation = "relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(32, (3, 3), padding="same", strides = 2, activation = "relu"))
    model.add(Activation("relu"))
    model.add(Flatten())
    model.add(Dense(192, activation="relu"))
    model.add(Dense(96, activation="relu"))
    model.add(Dense(48, activation="relu"))
    model.add(Dense(output_size, activation="linear"))

    adam = Adam(learning_rate=1e-5)
    model.compile(loss="mean_squared_error", optimizer=adam)

After completing the training the model has

  • loss (MSE) of 0.02 on "easy" tracks and >0.1 on harder ones (more detailed)
  • R^2 index of 0.7 on easy and <0.3 on hard

Because the performance of the CNN was not enough I also tried a Residual network. The model is far bigger (4 million parameters) and has slightly better results on detailed tracks:

  • loss (MSE) of ~0.03 on easy and ~0.05 on hard
  • R^2 index of 0.7 on easy and ~0.5 on hard

There isn't almost any difference, if not the plain CNN performing better, on simple tracks.

If you want the code for both models is here.

Questions

  • First of all, are there any errors in my code?

  • Could a larger dataset improve the quality of the predictions?

  • Could higher resolution images and/or RGB help? Could a different camera angle (eg camera set higher up) also help?

  • I generated dataset on different tracks and with "noisy" driving (swirling from left to right, breaking randomly, etc). Is this a good idea or does it just slow down the training?

  • In my opinion the sensors vector is (loosely) structured and values are significant to each other to some extent. Can this property be used?

  • Is there any other recommended architecture or strategy for this kind of regression with images?

Thanks in advance for any answer.

$\endgroup$
2
  • $\begingroup$ Hello. Welcome to AI SE! You're asking many questions in this post. Ideally, a post should contain only one main specific question. If you have sub-questions, that's fine, but try to keep it as simple as possible. If you have multiple questions, sometimes, it's just better to split the original post into multiple ones. $\endgroup$ – nbro Jun 5 at 12:45
  • $\begingroup$ Why use a net to estimate that sensor data in the first place? you might be able to get away with explicitly detecting the center line and doing math to figure out sensors[0] and sensors[20]. Same goes for the range finders. you could detect the edges, or do semantic segmentation to detect the road and use the edges of the road image. $\endgroup$ – juicedatom Jun 5 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.