Detecting license plate using tensorflow

Question

I'm currently working on license plate recognition. My system consist of 2 stage: (1) License Plate region extraction & (2) License Plate region recognition.

I'm doing (1) with Raspberry pi 3 model b. I find license plate candidate first by merging bounding boxes based on their similarity. In this way, i have only 1~7 license plate region proposals. And it took less than .3 seconds.

Now i have to reduce number of region proposal to be around only 1~2 so that i can send these images to server to do job (2). For license plate extraction, I made my own classifier function in tensorflow and the code is below. It gets proposed license plate as input.

First, I resize all license plate to be [120, 60] and converted to gray image. And there are 2 classes: 'plate', 'non_plate'. For non_plate image, i collected various image that might appear in image as background. I have 181 images for 'plate' class and 56 images for 'non_plate' for now, i trained for about 3000 steps so far and current loss is .53 .

When i did prediction on test set, i encountered problem that for some of plate image, it doesn't recognize license plate which is very obviously license plate image from my eyes. It is okay for me to wrongly recognize non plate image as plate but it is problem if it wrongly recognize plate as non_plate because it will not be sent to server to be fully recognized.

It happens like 10 out of 100 test images and this rate is far worse than i expected. I need help for adressing this problem. Would there be any improvement that i can make?

(1) Is my training set too small to classify between license plate and non license plate? Or is number of steps is too small?

(2) Is my graph structure bad? I needed to have small graph structure for my raspberry pi to recognize less than 1 second. Could you suggest better structure if it is bad?

(3) Is it bad to resize any proposed image to [120, 60] to be used as input for graph? I think it loses some information. But isn't this close to roi pooling like used in fast rcnn?

 inputs=tf.reshape(features[FEATURE_LABEL],[-1,120 , 60 ,1],name="input_node") #120 x 60 x 1, which is gray

conv1=tf.layers.conv2d(inputs=inputs,
                       filters=3,
                       kernel_size=[3,3],
                       padding='same',
                       activation=tf.nn.leaky_relu
                       )
#conv1 output shape: (batch_size,120,60,3)

pool1=tf.layers.max_pooling2d(inputs=conv1,pool_size=[2,2],strides=2,padding='valid')

#pool1 output shape: (batch_size,60,30,3)

conv2=tf.layers.conv2d(inputs=pool1,filters=6,kernel_size=[1,1],padding='same',activation=tf.nn.leaky_relu)

#conv2 output shape: (batch_size, 60,30,6)

pool2=tf.layers.max_pooling2d(inputs=conv2,pool_size=[2,2],strides=2,padding='valid')

#pool2 output shape: (batch_size, 30,15,6)

conv3=tf.layers.conv2d(inputs=pool2,filters=9,kernel_size=[3,3],padding='same',activation=tf.nn.leaky_relu)

#conv3 output shape: (batch_size, 30,15,9)

pool3=tf.layers.max_pooling2d(inputs=conv3,pool_size=[2,2],strides=2,padding='valid')

#pool3 output shape: (batch_size, 15,7,9)


#dense fully connected layer
pool2_flat=tf.reshape(pool3,[-1,15*7*9]) #flatten pool3 output to feed in dense layer

dense1=tf.layers.dense(inputs=pool2_flat,units=120,activation=tf.nn.relu)

logits=tf.layers.dense(dense1,2) #input for softmax layer

[training non plate image example] []4 [training plate image example. It is region proposed image]

Answer 1

Such a task is getting easier to complete, but there are still difficulties with rotations, skewing, scaling---to name a few issues. Your network has the benefit of simplicity and lightness for the target hardware, but it may suffer under the above conditions.

So 237 images (181+56) may be small for a "generic" approach, depending on how representative and diverse the dataset is. Also, the dataset is unbalanced (the first class has twice as many examples), which causes bias in learning.

There are several ways to expand a base dataset:

• Transform images and add them (with the same label, if supervised learning) to the dataset. Many libraries allow to rotate, skew, scale, or even blur images. Be careful as transforming needs be "reasonable", and changing image format over and over creates artifacts that can confuse the machine (e.g. too much JPEG savings on transformed images).
• Generate synthetic data. Assuming license plates have a known format, it may be easy to generate images with good fidelity to real plates. This is not always possible, but license plates are formalized, so there should be only a handful patterns (typically plain, light, diplomatic, military vehicles).

Aside the dataset (potential) issue, the graph is fine. However, it may be worth trying different settings. It really depends on how you have ended up with the current graph. Removing pooling layers helps keeping more information, as well as a larger input image. 120x60 looks pretty small, and the comparison to fast-RCNN's RoI pooling layer looks odd, given that RoI comes after the feature maps. So a larger input image could give better results.