How can max-pooling be applied to find features in words?

Question

I'm reading about max-pooling in a dynamic CNN paper. I can see how it can help find features in images, given that the pixel with the highest density gets pooled, but how does it help to find features in words?

Answer 1

In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.

For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others.

MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.

When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is searching for a feature, and so you use an exorbitant number of the kernels at various sizes to search for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used.

A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).