Which neural network to use for optical mark recognition?

Question

I've created a neural net using the ConvNetSharp library which has 3 fully connected hidden layers. The first having 35 neurons and the other two having 25 neurons each, each layer with a ReLU layer as the activation function layer.

I'm using this network for image classification - kinda. Basically it takes inputs as raw grayscale pixel values of the input image and guesses an output. I used stochastic gradient descent for the training of the model and a learning rate of 0.01. The input image is a row or column of OMR "bubbles" and the network has to guess which of the "bubble" is marked i.e filled and show the index of that bubble.

I think it is because it's very hard for the network to recognize the single filled bubble among many.

Here is an example image of OMR sections:

Using image-preprocessing, the network is given a single row or column of the above image to evaluate the marked one.

Here is an example of a preprocessed image which the network sees:

Here is an example of a marked input:

I've tried to use Convolutional networks but I'm not able to get them working with this.

What type of neural network and network architecture should I use for this kind of task? An example of such a network with code would be greatly appreciated.

I have tried many preprocessing techniques, such as background subtraction using the AbsDiff function in EmguCv and also using the MOG2 Algorithm, and I've also tried threshold binary function, but there still remains enough noise in the images which makes it difficult for the neural net to learn.

I think this problem is not specific to using neural nets for OMR but for others too. It would be great if there could be a solution out there that could store a background/template using a camera and then when the camera sees that image again, it perspective transforms it to match exactly to the template

I'm able to achieve this much - and then find their difference or do some kind of preprocessing so that a neural net could learn from it. If this is not quite possible, then is there a type of neural network out there which could detect very small features from an image and learn from it. I have tried Convolutional Neural Network but that also isn't working very well or I'm not applying them efficiently.

Answer 1

From what I understand, don't bother with a CNN, you have essentially perfectly structured images.

You can hand code detectors to measure how much filled in a circle is.

Basically do template alignment and then search over the circles.

Ex a simple detector would measure the average blackness of the circle which you could then threshold.

Answer 2

I'm not familiar with ConvNetSharp library, and the tag convolutional-neural-networks is a bit confusing me, but from :

So I've created a neural net using the ConvNetSharp library which has 3 fully connected hidden layers. The first having 35 neurons and the other two having 25 neurons, each with a ReLU layer as the activation function layer.

I assume you are building just a densely connected neural network. Correct me if I'm wrong.

---

The type of neural network you need is Convolutional Neural Network.

For image recognition (which is your case), convolutional network are almost always the answer.

There is plenty of type of CNN, just pick one that seems appropriate and try.

In my opinion, your task seems quite simple, you won't need really deep / complex architecture.

---

It would be great if there could be a solution out there that could store a background/template using a camera and then when the camera sees that image again

I am not aware of a model that could do what you are asking.

But what you are asking is not really in the 'neural network' mindset. The goal of building a neural network is that you don't specify anything. The neural network will learn and find the features for you. So you just have to feed him a lot of data, and he will be able to recognize your pattern.

Take a look at this visualization of CNN filters :

Here, no one gave the neural network the template of a nose or the template of an eye or the template of a face. The CNN learned it over a lot of image.

Answer 3

Is it the case that one of the numbers if filled in? If so a CNN with 10 output should work well. Just choose the output that has the highest probability. If your data allows no number to be filled in, then have 11 outputs where the eleventh output indicates none is filled in. I would recommend transfer learning using the MobileNet model. Documentation is here. Here is the code to adapt MobileNet to your problem:

image_size=128
no_of_classes=10  # set to 11 if in some cases no numbers are filled in
lr_rate=.001
dropout=.4
mobile = tf.keras.applications.mobilenet.MobileNet( include_top=False,
                                                           input_shape=(image_size,image_size,3),
                                                           pooling='avg', weights='imagenet',
                                                           alpha=1, depth_multiplier=1)
x=mobile.layers[-1].output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x=Dense(128, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
                bias_regularizer=regularizers.l1(0.006) ,activation='relu')(x)
x=Dropout(rate=dropout, seed=128)(x)
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (no_of_classes, activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)    
for layer in model.layers:
    layer.trainable=True
model.compile(Adam(lr=lr_rate), loss='categorical_crossentropy', metrics=['accuracy']) 

I would also create a training, test and validation set with about 150 images in the test set and 150 images in the validation set leaving 1700 images for training. I also recommend you use two useful callbacks. Documentation is here. Use the ReduceLROnPlateau callback to monitor the validation loss and adjust the learning rate downward by a factor. Use ModelCheckpoint to monitor the validation loss and save the model with the lowest validation loss to use to make predictions on the test set.