What can be an example for the prior knowledge used in Deep Learning systems?

Question

It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from 1.1 The deep learning revolution of the textbook named Deep learning with PyTorch by Eli Stevens, Luca Antiga, Thomas Viehmann, regarding the role of feature engineering in deep learning

Deep learning, on the other hand, deals with finding such representations automatically, from raw data, in order to successfully perform a task. In the ones versus zeros example, filters would be refined during training by iteratively looking at pairs of examples and target labels. This is not to say that feature engineering has no place with deep learning; we often need to inject some form of prior knowledge in a learning system. However, the ability of a neural network to ingest data and extract useful representations on the basis of examples is what makes deep learning so powerful. The focus of deep learning practitioners is not so much on handcrafting those representations, but on operating on a mathematical entity so that it discovers representations from the training data autonomously. Often, these automatically created features are better than those that are handcrafted! As with many disruptive technologies, this fact has led to a change in perspective.

The paragraph clearly saying that we need to inject some form of prior knowledge into the learning system. What can be a concrete example for such prior knowledge we are used in deep learning systems?

Answer 1

I would distinguish at least 2 cases when it comes to a generic expression like prior knowledge:

• generic extra information provide to a model, really close if not the same as feature engineering.
• literal prior probability distributions used to initialize or guide a model during training.

For the first case there's plenty of examples that we can provide. The most intuitive maybe is use of masks in computer vision. Let's say we want to clean an image (i.e. haze removal) as a pre processing step for a self driving car system. Then we could train a model and feed to it not only the image captured by the camera, but also a depth mask estimated using another model. In this case the other model works as a prior distribution, since the model is not learning it, it just leverage that extra information that comes with the image.

For the second case, there are specific class of models that learn and sometimes require prior distributions as an input, the most known to me are Bayesian Neural Networks.

Why bothering providing a prior for these models? Well, there are at least 2 reasons: sometimes we have information about a system we're trying to describe, so we can make the training more efficient, for example we're might trying to fit a coin toss model, but we know the coin is not fair and that the resulting probabilities returned by the model should not be 50/50. The second reason is that sometimes we also want to train models that do not return only raw probabilities, but also estimate the uncertainty level of those probabilities. To do that, the model learns a posterior probability over the data, and it does that by updating an initial prior. Note that the prior for this class of models can also be randomly initialize.