How should the racing agent take into account the velocity of the vehicle, given the images with a speedometer?

Question

I'm developing a game AI, which tries to master racing simulations. I already trained a CNN (AlexNet) on in-game footage of me playing the game and the pressed keys as the target. As the CNN is only making predictions on a frame-to-frame basis, and I resized the image input to 160x120 due to GPU memory limitations, it cannot read the speedometer, so it seems not to have a feeling for its current velocity.

I thought of different ways to fix this issue:

1. Crop the captured image down to the size of the speedometer, which displays the current speed in mph, and feed the low-resolution game image, as well as the relatively high-res image (70x30) of the current speed into the neural network, which makes predictions based on the two images.

2. As I don't know whether AlexNet can serve as an OCR as well, my second thought was to use an existing one (like tesseract-ocr/PyTesser) on the cropped image and feed its output to the fully connected layer.

3. I already tried to implement an optical-flow algorithm, but, sadly, non of the Python ones seems to output good real-time results. I wonder whether I can input the current frame as well as the last one, and let AlexNet figure out the movement.

As the processing has to happen in real-time, and the only performance reviews of pytesser I found reported a processing time of ~100ms (never tested that).

My question is: what method would work best?

The optical flow approach would have the advantage of the AI knowing in which direction other cars are moving as well.

Answer 1

Option 1 would be a very interesting one from a research perspective. I cannot imagine that CNNs have to capacity at the moment, to learn the concept of numbers and apply them in a useful way. If it was an analog speedometer, things would be different. But it would be really interesting to try it and see what you can achieve with this approach. I haven't read any research papers yet, where such a challenge would have been mastered.

If you are less concerned with research but rather get this project to work, I would propose an approach similar to option 2. The OCR tools you mentioned were designed to identify symbols, even if distorted or otherwise hard to read. In your case, the numbers will always look the same and are most likely always in fixed positions. Therefore, using fancy OCR algorithms or neural nets are overkill for the problem at hand. You can write a simple algorithm that crops the speedometer, searches for the 10 possible patterns (0 to 9) with 10 specific kernels and calculate the speed from the result. This can be implemented efficiently without the need to train a CNN or some other complex algorithm.

---

Method for implementation:

To answer the first question in the comments. The algorithm for this is really easy. Your image will be represented by a 2D array (or maybe a 3D array if you are working with colors). All you need to do is find out, what your numbers look like and store the appropriate array. You should end up with 10 different arrays for 10 different numbers.

To find out if a number is in your current picture, you only have to check if the array for this number is a subset of the array of your current picture. The position, where the arrays are matching, will also indicate the position of the number. You do this for all 10 numbers and can calculate the actual value displayed on the speedometer afterwards from the results.

This is very easy if the numbers always look exactly identical. In this case you can simply look for an exact match. If there are slight variations, you can look for parts of the image array that is similar to your template arrays. Just sum the absolute values of the diffs of each pixel value and if it is below a certain threshold, you have your match. That's all there is to it.

In python I like working with PIL for this kind of image manipulation, but there are other frameworks out there as well.