# How to recognize sequence of digits in an image

I am learning to program neural networks and others, and I would like to know how I can get the numbers that are in an image, for example if I pass an image that has 123 written, get with my model that there are 123 written, I have tried to use PyTesseract is not very precise, and I would like to do it with a neural network, my current code is quite simple, it recognizes the digits of the mnist dataset such that:

import tensorflow as tf
from tensorflow.keras import Sequential, optimizers
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D
import matplotlib.pyplot as plt

mnist = tf.keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

print('train_images.shape:', train_images.shape)
print('test_images.shape:', test_images.shape)
plt.imshow(train_images)

train_images = train_images.reshape((60000, 28, 28, 1))
test_images = test_images.reshape((10000, 28, 28, 1))

train_images = train_images.astype('float32') / 255
test_images = test_images.astype('float32') / 255

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

model = Sequential()

model.add(Conv2D(32, (5, 5), activation = 'relu', input_shape = (28, 28, 1)))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (5, 5), activation = 'relu'))
model.add(MaxPooling2D((2, 2)))

model.add(Flatten())

model.add(Dense(10, activation = 'softmax'))

model.summary()

model.compile(loss = 'categorical_crossentropy', optimizer = 'sgd', metrics = ['accuracy'])

model.fit(train_images, train_labels, batch_size = 100, epochs = 5, verbose = 1)

test_loss, test_accuracy = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_accuracy)


but I would need to know how I can pass an image with a sequence of digits to it, and that it recognizes the digits in question, does anyone know how I could do it? Thanks a lot.

• I don't know exactly what you're looking for and I didn't read your post carefully, but have you heard of optical character recognition? If not, that's probably something you want to look into. – nbro Jan 25 at 12:29
• @nbro Yes, it's what I'm using right now, but I can't make it very accurate – John Doe Jan 26 at 17:23
• @JohnDoe your code is image classification, won't do object detection of digits – datdinhquoc Jan 27 at 8:14

## 2 Answers

Your task is text recognition, however your code is for classification task. So you need to use different approach for that. You mentioned that you're going to give model 123 and get 123. But you can not do that with just convolutional networks. Images with text are sequential, so you need to use CRNN(Convolutional-Recurrent-Neural-Networks), LSTM(Long-Short-Term-Memory), BiLSTM(Bidirectional-LSTM). In most of the research papers there are convolutional networks are being used just for feature extraction stage. For prediction stage they used recurrent units, such as LSTM cells or RNN.

• This might be true generally, but I suspect it is not true in the OP case. If you were looking for words, a recurrent or LSTM network would make sense. The OP is looking for digits. There is no indication in the question that these follow any pattern, so should be viewed as random values. – David Hoelzer Jan 27 at 11:47

From your question there is no indication that there is any pattern to these digits. If there were, the recommendation for an LSTM or RCNN would make sense. In the case of random values, I have found that a two or three layer CNN that then descends through two parallel dense networks does an excellent job identifying CAPTCHA style random characters. One path is primarily responsible for identifying bounding boxes, the other is determining which characters are present.

There are many other ways to solve this problem. You might do well to research CAPTCHA solving via neural networks. More specifically, text based CAPTCHAs. If your task really is just OCR, then research OCR through neural networks. The technique I describe here will work but will become cumbersome for a page of text, for example. In that case a sliding window CNN coupled with a dense layer and an LSTM makes the most sense since you will be dealing with predictable sequences of characters.