I'm developing a Game AI which tries to master racing simulations. I already trained a CNN (alexnet) on ingame footage of me playing the game and the pressed keys as the target. As the CNN is only making predictions on a frame-to-frame basis, and I resized the image input to 160x120 due to GPU memory limitations, it cannot read the speedometer, thus seems not to have a feeling for its current velocity. I thought of different ways to fix this issue:
Crop the captured image down to the size of the speedometer, which displays the current speed in mph, and feed the low resolution game image, as well as the relatively high-res image (70x30) of the current speed into the neural network, which makes predictions based on the two images.
As I don't know whether alexnet can serve as an OCR as well, my second thought was to use an existing one (like tesseract-ocr/PyTesser) on the cropped image and feed its output to the fully connected layer.
I already tried to implement an optical-flow algorithm, but sadly, non of the python ones seems to output good real-time results. I wonder whether I can input the current frame as well as the last one, and let alexnet figure out the movement.
As the processing has to happen in real time, and the only performance reviews of pytesser I found reported a processing time of ~100ms (never tested that). My Question is, what method would work best.
Edit: Optical flow would have the advantage of the AI knowing in which direction other cars are moving as well.