How to detect LEGO bricks by using a deep learning approach?

Question

In my thesis I dealt with the question how a computer can recognize LEGO bricks. With multiple object detection, I chose a deep learning approach. I also looked at an existing training set of LEGO brick images and tried to optimize it.

My approach

By using Tensorflow's Object Detection API on a dataset of specifically generated images (Created with Blender) I was able to detect 73.3% of multiple LEGO Bricks in one Foto.

One of the main problems I noticed was, that I tried to distinguish three different 2x4 bricks. However, colors are difficult to distinguish, especially in different lighting conditions. A better approach would have been to distinguish a 2x4 from a 2x2 and a 2x6 LEGO brick.

Furthermore, I have noticed that the training set should best consist of "normal" and synthetically generated images. The synthetic images give variations in the lighting conditions, the backgrounds, etc., which the photographed images do not give. However, when using the trained Neural Network, photos and not synthetic images are examined. Therefore, photos should also be included in the training data set.

One last point that would probably lead to even better results is that you train the Neural Network with pictures that show more than one LEGO brick. Because this is exactly what is required by the Neural Network when it is in use.

• Are there other ways I could improve upon this?

(Can you see any further potential for improvement for the Neural Network? How would you approach the issue? Do any of my approaches seem poor? How do you solve the problem?)

Answer 1

So I am assuming that you are trying to detect a lego brick from the image. One idea is that you can use transfer learning. Leveraging a pre-trained machine learning model is called transfer learning. The underlying idea behind transfer learning is that one takes a well-trained model from one dataset or domain, and applies it to a new one. François Chollet has written a very comprehensive guide to transfer learning (https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)

I admit I took some of this information from Christopher Bonnett's article named Classifying e-commerce products based on images and text.

I also suggest using the Lego brick dataset from Kaggle on this link: https://www.kaggle.com/joosthazelzet/lego-brick-images It has over 12,700 lego brick images.

If processing power is a problem, you can use Amazon Web Services for cloud computing. It is inexpensive for small scale operations like this.

Of course for the object detection, you can always increase the number of convolution layers. However, if you have too many layers, you should also include residual blocks/residual network. This would allow neural networks even with over a thousand layers to operate effectively. This video should help you understand how residual networks work (https://www.youtube.com/watch?v=ahkBkIGdnWQ)

Finally, make sure not to overfit during your training and if you do follow the residual network idea, you should also include upsampling in your convolution neural network( More in here: https://towardsdatascience.com/up-sampling-with-transposed-convolution-9ae4f2df52d0)

I hope this helped and good luck on your endeavor.