Generate credit cards dataset for locating number region

Question

Currently I'm working on a project for scanning credit card and text extraction from cards. So first of all I decided to preprocess my images with some filters like thresholding, dilation and some other stuff. But it was not successfully for OCR of every credit cards. So I learned a lot and I found a solution like this for number plate recognition that is very similar to my project. In the first step I want to generate a random dataset like my cards to locate card number region, and for every card that I've generated I cropped two images that one of them has numbers and another has not. I generated 2000 images for every cards.

so I have some images like this:

(does not have numbers) (has numbers)

And after generating my dataset I used this model with tensorflow to train my network.

    model = models.Sequential()
    model.add(layers.Conv2D(8, (5, 5), padding='same', activation='relu', input_shape=(30, 300, 3)))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(16, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(32, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Flatten())
    model.add(layers.Dense(512, activation='relu'))
    model.add(layers.Dense(2, activation='softmax'))

Here is my plot for 5 epochs.

I almost get 99.5% of accuracy and It seems to be wrong, I think I have kind of overfitting in my data. Does it work correctly or my model is overfitted ? And how can I generate dataset for this purpose ?

Answer 1

I am assuming the question you are asking is how to prevent over-fitting on the maximum accuracy. Your graph does show that your model over-fits.

There is a couple of different methods to prevent over-fitting from happening. You can specify training to stop after a certain amount of epochs. In your case it seems to be 2 or 3 epochs. Take care as a new initialization might require more or less epochs to reach optimal accuracy thus it is required run your model a couple of times to determine the correct amount of epochs.

You can also specify the accuracy you want your model to reach before you stop training, this can be dangerous as a local-minima could be found below your expected accuracy thus resulting in your training to run indefinitely. Also your model could have become more accurate then your expected accuracy but stopped prematurely due to the model reaching your stopping-condition.

You can combine the two. Terminate training once it reach your expected accuracy otherwise terminate it if it reaches a certain amount of epochs. This way you prevent your training from happening indefinitely.

In Tensorflow you would want to build your own custom training seen here

The method you are interested is

epoch_accuracy = tfe.metrics.Accuracy()

In their example they get the accuracy after each epoch, you would want to create your own batches and apply it there.

If you really want to dive deeper, you can implement methods for the model to detect when it starts over-fitting such as looking at the standard deviation of the accuracy across a certain measure (usually epochs but your model starts over-fitting after 5 so its not a good measure in your case, maybe do standard deviation over a certain amount of batches trained) if the accuracy leaves the standard deviation space then it might be good idea to stop training. The problem with this technique is that your model has to over-fit a bit to know it needs to stop. The other problem is that your model might seem it is over-fitting but was just a dip in accuracy and was stopped prematurely. (There is methods around this but can't recall them)

Another technique is to add drop out. This introduce noise and pushes your model out of a local-minima. Also forces nodes that was ignored to be used and optimized. This surprisingly works well but is not a surefire way to prevent over-fitting. This technique is built into Tensorflow and don't need custom training. You can find more reading about Tensorflow dropout here, also they have more techniques on how to overcome over-fitting but their solutions is to change the model and your model seems to be fine.

tf.keras.layers.Dropout(0.2),

These are just some techniques I listed but there is a lot more out there. Unfortunately there is no concrete method to prevent it but steps can be taken to stop well before over-fitting occurs. The first two techniques are mostly used together.