I'm developing a game AI, which tries to master racing simulations. I already trained a CNN (AlexNet) on in-game footage of me playing the game and the pressed keys as the target. As the CNN is only making predictions on a frame-to-frame basis, and I resized the image input to 160x120 due to GPU memory limitations, it cannot read the speedometer, so it seems not to have a feeling for its current velocity.
I thought of different ways to fix this issue:
Crop the captured image down to the size of the speedometer, which displays the current speed in mph, and feed the low-resolution game image, as well as the relatively high-res image (70x30) of the current speed into the neural network, which makes predictions based on the two images.
As I don't know whether AlexNet can serve as an OCR as well, my second thought was to use an existing one (like tesseract-ocr/PyTesser) on the cropped image and feed its output to the fully connected layer.
I already tried to implement an optical-flow algorithm, but, sadly, non of the Python ones seems to output good real-time results. I wonder whether I can input the current frame as well as the last one, and let AlexNet figure out the movement.
As the processing has to happen in real-time, and the only performance reviews of pytesser I found reported a processing time of ~100ms (never tested that).
My question is: what method would work best?
The optical flow approach would have the advantage of the AI knowing in which direction other cars are moving as well.