1
$\begingroup$

I'm reading about max pooling in a dynamic CNN paper and I can see how it can help find features in images cause the pixel with the highest density gets pooled, but how does it help find features in words?

$\endgroup$
0
$\begingroup$

In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.

For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others.

MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.

When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is searching for a feature, and so you use an exorbitant number of the kernels at various sizes to search for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used.

A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).

| improve this answer | |
$\endgroup$
  • $\begingroup$ see that makes sense for images, but I saw a paper using it for words which didn't make sense to me $\endgroup$ – user8714896 Nov 21 '19 at 3:02
  • $\begingroup$ I gave the example across whole temporal dim but maybe imagine also using a finite region— which maintains tempora correlations and fits the idea like images $\endgroup$ – mshlis Nov 21 '19 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.