2
$\begingroup$

It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from 1.1 The deep learning revolution of the textbook named Deep learning with PyTorch by Eli Stevens, Luca Antiga, Thomas Viehmann, regarding the role of feature engineering in deep learning

Deep learning, on the other hand, deals with finding such representations automatically, from raw data, in order to successfully perform a task. In the ones versus zeros example, filters would be refined during training by iteratively looking at pairs of examples and target labels. This is not to say that feature engineering has no place with deep learning; we often need to inject some form of prior knowledge in a learning system. However, the ability of a neural network to ingest data and extract useful representations on the basis of examples is what makes deep learning so powerful. The focus of deep learning practitioners is not so much on handcrafting those representations, but on operating on a mathematical entity so that it discovers representations from the training data autonomously. Often, these automatically created features are better than those that are handcrafted! As with many disruptive technologies, this fact has led to a change in perspective.

The paragraph clearly saying that we need to inject some form of prior knowledge into the learning system. What can be a concrete example for such prior knowledge we are used in deep learning systems?

$\endgroup$
1
  • $\begingroup$ Here is a related (if not duplicate) question. $\endgroup$
    – nbro
    Oct 8 at 12:23
1
$\begingroup$

I would distinguish at least 2 cases when it comes to a generic expression like prior knowledge:

  • generic extra information provide to a model, really close if not the same as feature engineering.
  • literal prior probability distributions used to initialize or guide a model during training.

For the first case there's plenty of examples that we can provide. The most intuitive maybe is use of masks in computer vision. Let's say we want to clean an image (i.e. haze removal) as a pre processing step for a self driving car system. Then we could train a model and feed to it not only the image captured by the camera, but also a depth mask estimated using another model. In this case the other model works as a prior distribution, since the model is not learning it, it just leverage that extra information that comes with the image. enter image description here

For the second case, there are specific class of models that learn and sometimes require prior distributions as an input, the most known to me are Bayesian Neural Networks. enter image description here

Why bothering providing a prior for these models? Well, there are at least 2 reasons: sometimes we have information about a system we're trying to describe, so we can make the training more efficient, for example we're might trying to fit a coin toss model, but we know the coin is not fair and that the resulting probabilities returned by the model should not be 50/50. The second reason is that sometimes we also want to train models that do not return only raw probabilities, but also estimate the uncertainty level of those probabilities. To do that, the model learns a posterior probability over the data, and it does that by updating an initial prior. Note that the prior for this class of models can also be randomly initialize.

$\endgroup$
2
  • $\begingroup$ This answer provides good information, but it mixes concepts, so it may be confusing. For example, you say that the second way to interpret "prior knowledge" is "literal prior probability distributions used to initialize or guide a model during training", but then you give the example of "Bayesian neural networks". It's true that BNNs are based on having distributions over the weights, but in your second point you mention probability distributions to initialize or guide a model during training, which seemed to suggest that you were referring to ways to initialise the weights. $\endgroup$
    – nbro
    Oct 8 at 12:29
  • $\begingroup$ Note that I am not saying that BNNs are not a good example where we can introduce prior knowledge. In fact, in BNNs, we can really specify priors for the weights (as priors in the Bayes' rule, which are used to introduce "prior knowledge"), but it just seems that you were referring to the initialization of the weights and other techniques to guide the training of non-Bayesian neural networks. Another thing you may want to point out is the relationship between "prior knowledge" and "inductive bias" (and/or model architecture). $\endgroup$
    – nbro
    Oct 8 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.