# Recognition of lines in a chalkboard

I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it.

It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...

After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.

Any hint ?

Input will be a sequence as this one, this one or this one (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.

Thanks.

Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references.

• I would look into Hough transform en.wikipedia.org/wiki/Hough_transform Dec 14 '19 at 18:53
• @GeorgeWhite: thanks for your comment. As you know, Hough transform is mainly to detect straight lines. It can be generalized to some other parametric curves, but I do not see promising use it in the case of free forms. Dec 14 '19 at 19:31
• From ""start of line at xxx,xxx", "continue line at xxx,xxx"" I thought you were assuming straight lines. Dec 14 '19 at 19:33

I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.

Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line".

One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index idx and the one right after, at index idx+1 (captured at instants $$t$$ and $$t + \Delta t$$, where $$\Delta t$$ is the sampling interval).

At coordinates $$(i,j)$$, this derivative could look something like: timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.

Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.

When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.

This is the main idea, which can be improved with:

• Defining the area of the blackboard, either by hand or automatically
• Thresholding the images to better isolate the chalk trace from the blackboard
• Using a tracker such as a Kalman filter to know where to look next for the chalk
• Thanks for your answer. I think movement of the teacher in front of the chalkboard will cause hard interferences to your proposal of algorithm. Dec 15 '19 at 19:52
• Indeed. To cut out the teacher, you can threshold then apply a blob extraction algorithm, such as the two pass algorithm. This can identify the teacher if done correctly. You can then cut the teacher out of the image and color his pixels like the blackboard. This comes of course before the start of what I wrote above. Dec 15 '19 at 21:37

Maybe Just use simple Convnets (Pre-trained perhaps) and train it on the images of the teacher on the blackboard. You could use a GAN to remove the teacher and complete the rest of the image (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwj9lKKSh43uAhWbfH0KHeaDC1UQFjAAegQIAxAC&url=http%3A%2F%2Fstanford.edu%2Fclass%2Fee367%2FWinter2018%2Ffu_guan_yang_ee367_win18_report.pdf&usg=AOvVaw2tG3bStRlys_NPLX9-XPep) But that would be too troublesome.

The best way would be to take real-time video chunks, use a Convolutional Network to detect the shapes you want, and return bounding boxes for the appropriate shapes (for the location purposes) and their 'classification' - whether a straight line, curved line, or some other user defined shape. You could also choose to use some other YOLO (You Only Look Once) technique. You can check out with: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAJegQIARAC&url=https%3A%2F%2Ftowardsdatascience.com%2Fobject-detection-using-deep-learning-approaches-an-end-to-end-theoretical-perspective-4ca27eee8a9a&usg=AOvVaw2TkW094_O5Q7-mcVMJ5SEN ;

You also won't have to deal with the pesky teacher with the above method (assuming he doesn't stand in one place and constantly block a part of the BB). These methods are near SOTA and would be extremely effective than conventional algos. Not to mention that using Keras is a piece of cake with a giant community and endless resources to help you in case you get stuck on some problem. Since it is easy to use, you can setup a prototype in almost no time.

• please, recall note at the end of the question Jan 9 '21 at 15:54
• @pasabaporaqui You want someone to implement the code for you? Apart from that, the only details I can add in my answer is an explanation on CNN's, if you want. Jan 9 '21 at 17:11
• What I do not want, as the note says, is a empty "use a GN" or "use a CNN" followed by a useless list of 3 links explaining the generics of Deep Learning or object detection. Instead, it is expected a reference to a paper, algorithm or method, something as "in xxx work xxx shows how he/she has combined the Hough transform with a CNN to obtain the xxx result in a scenario similar to the one of the question". Jan 9 '21 at 20:04