Is there any way to control the extraction of features? How do I determine which features are been learned during training, i.e relevant information is been learned or not?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
0
There are methods called "scoring systems" where you give a image scores such as "0.9 stripes, 0.0 red, 0.8 hair, ..." and use those scores to classify objects. It's an older idea, not used to determine if the network is learning. It's not in a standard CNN.
To determine if relevant information is being learned or not, it's standard to use the testing accuracy, training accuracy, confusion matrix, or AUC.
Determining what exactly a CNN is learning is a complicated research problem that's ongoing. In short - you can't really know. For a basic network, you can tell that it is learning something but not what it's actually using to make determinations.
$\begingroup$
$\endgroup$
Yes there is! If a model generalizes well to the test set, we already know that it has found some useful features. However, the latent representation of the data may still be "entangled" - a single element of the latent vector may actually encode information about multiple attributes of the input, or a single attribute may be spread across multiple elements. We usually prefer a representation in which the features are represented by the axes of the latent space - a "disentangled" representation. For example, if we were encoding faces, it would be nice to have an axis for smiling/not, another for masculine/feminine, and ao on.
Pushing models to learn "clean" (disentangled) representations is an active sub-field of machine learning research with practical applications (like interpretability, but also because it makes it easier for "downstream" models to learn their tasks, e.g. a control policy in reinforcement learning system taking as input a learned representation from a world model).
Where to begin? Start with L2 regularisation to push your network to "spend" those weights wisely (more weights close to zero => sparser latent vector) and work your way up from there.
$\begingroup$
$\endgroup$
So there's pictures of low level activation maps, and some gradient based information where yoy take the deriviative of the output with respect to the input and generate a heatmap.
I kind of have my doubts on how usefull this is in general, imo it kind of is creating a fallacious illusion of understanding.
There's some additional research using blurring to figure out the relevant features also but again I have my doubts.
Probably the most usefull is generating images by optimizing your class score. You can learn how badly your CNN actually labels things(doing this makes you realize quickly that CNN are garbage at actually understanding and incredibly easy to trick).
