# Neural Network for Optical Mark Recognition?

I've created a neural net using the ConvNetSharp library which has 3 fully connected hidden layers. The first having 35 neurons and the other two having 25 neurons each, each layer with a ReLU layer as the activation function layer.

I'm using this network for image classification - kinda. Basically it takes inputs as raw grayscale pixel values of the input image and guesses an output. I used stochastic gradient descent for the training of the model and a learning rate of 0.01. The input image is a row or column of OMR "bubbles" and the network has to guess which of the "bubble" is marked i.e filled and show the index of that bubble.

I think it is because it's very hard for the network to recognize the single filled bubble among many.

Here is an example image of OMR sections:

Using image-preprocessing The network is given a single row or column of the above image to evaluate the marked one.

Here is an example of a preprocessed image which the network sees:

Here is an example of a marked input:

I've tried to use Convolutional networks but I'm not able to get them working with this.

Basically, my question is that what type of neural network and network architecture should I use for this kind of task?. An example of such a network with code would be greatly appreciated.

I have tried many preprocessing techniques such as background subtraction using the AbsDiff function in EmguCv and also using the MOG2 Algorithm and I've also tried threshold binary function but there still remains enough noise in the images which makes it difficult for the neural net to learn.

I think this problem is not specific to using neural nets for OMR but for others too. It would be great if there could be a solution out there that could store a background/template using a camera and then when the camera sees that image again, it perspective transforms it to match exactly to the template

I'm able to achieve this much - and then find their difference or do some kind of preprocessing so that a neural net could learn from it. If this is not quite possible, then is there a type of neural network out there which could detect very small features from an image and learn from it. I have tried Convolutional Neural Network but that also isn't working very well or I'm not applying them efficiently.

• What is the structure of your dataset(any bounding boxes/meta data?). Also, how many images are you using in your dataset? – Udayraj Deshmukh Jul 5 '19 at 16:01
• @UdayrajDeshmukh The dataset consists of a little more than 2,000 preprocessed images (as shown above) that have been labeled. No, there is no such bounding box or metadata to go with the dataset. – Hamza Abdullah Jul 9 '19 at 12:07
• So there are only cropped vertical strips? Also is applying ML a necessity to solve your problem? – Udayraj Deshmukh Jul 9 '19 at 12:24
• @UdayrajDeshmukh Sorry for the late replies. Initially, the image comes in as a full document, it is then preprocessed and cropped to these vertical strips of options/bubbles. The reason why I must use ML is the fact that the sheet is arbitrary, it can be of any color, design and the bubbles can be of any size or shape, all of this depends on the user and in the program I've provided the user with an interface to train their models. I hope that answers your question. – Hamza Abdullah Jul 10 '19 at 13:38
• So if OMR Checking is your problem statement. It's coincidence that I have been working on a solution for the same, and it works pretty well for document scanned images currently – Udayraj Deshmukh Jul 10 '19 at 18:34

From what I understand, don't bother with a CNN, you have essentially perfectly structured images.

You can hand code detectors to measure how much filled in a circle is.

Basically do template alignment and then search over the circles.

Ex a simple detector would measure the average blackness of the circle which you could then threshold.

I'm not familiar with ConvNetSharp library, and the tag convolutional-neural-networks is a bit confusing me, but from :

So I've created a neural net using the ConvNetSharp library which has 3 fully connected hidden layers. The first having 35 neurons and the other two having 25 neurons, each with a ReLU layer as the activation function layer.

I assume you are building just a densely connected neural network. Correct me if I'm wrong.

The type of neural network you need is Convolutional Neural Network.

For image recognition (which is your case), convolutional network are almost always the answer.

There is plenty of type of CNN, just pick one that seems appropriate and try.

In my opinion, your task seems quite simple, you won't need really deep / complex architecture.

It would be great if there could be a solution out there that could store a background/template using a camera and then when the camera sees that image again

I am not aware of a model that could do what you are asking.

But what you are asking is not really in the 'neural network' mindset. The goal of building a neural network is that you don't specify anything. The neural network will learn and find the features for you. So you just have to feed him a lot of data, and he will be able to recognize your pattern.

Take a look at this visualization of CNN filters :

Here, no one gave the neural network the template of a nose or the template of an eye or the template of a face. The CNN learned it over a lot of image.

• Hi Astariul, I've tried using the convolutional neural network approach, but the results are not good for some reason. Maybe, its because the size of images are so small that the network isn't able to extract features and learn from them. – Hamza Abdullah Oct 20 '18 at 16:31
• I've updated the question to clarify things further, thanks for your consideration. – Hamza Abdullah Oct 20 '18 at 16:47

Is it the case that one of the numbers if filled in? If so a CNN with 10 output should work well. Just choose the output that has the highest probability. If your data allows no number to be filled in, then have 11 outputs where the eleventh output indicates none is filled in. I would recommend transfer learning using the MobileNet model. Documentation is here. Here is the code to adapt MobileNet to your problem:

image_size=128
no_of_classes=10  # set to 11 if in some cases no numbers are filled in
lr_rate=.001
dropout=.4
mobile = tf.keras.applications.mobilenet.MobileNet( include_top=False,
input_shape=(image_size,image_size,3),
pooling='avg', weights='imagenet',
alpha=1, depth_multiplier=1)
x=mobile.layers[-1].output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x=Dense(128, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
bias_regularizer=regularizers.l1(0.006) ,activation='relu')(x)
x=Dropout(rate=dropout, seed=128)(x)
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (no_of_classes, activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)
for layer in model.layers:
layer.trainable=True