3
$\begingroup$

I have read some articles where people use 1D CNN for NLP tasks like sentiment analysis. My questions are, given that CNNs are largely used for images, how/why does this work on sequences/NLP tasks? And, are there any specific CNN-based architectures that outperform transformers?

$\endgroup$

1 Answer 1

2
$\begingroup$

You can conceive images as 2D or 3D sequences of pixels. In the same way, text without any preprocessing or feature extraction is just a 1D sequence of words/characters. So there is no difference at all in the way CNN learn features from images or text (or any other type of data). Of course the limitations are also the same. For example in terms of linguistic dependencies, a single convolutional layer can learn relationships between words that fall into the same kernel window. In the example below a kernel of size 3 is used, so only dependencies between words/characters that fall within that window will be captured.

Regarding the comparison with transformer, there's a recent paper that compare CNN with depthwise scalable convolutions and transformers, showing that when finetuning pretrained models they both share similar performances, with convolutions being better in term of training speed and sequence length scalability.

As a last note, keep in mind that transformers have a big design difference compared to CNN, which is the explicit learning of positional encoding of features. A CNN learn kernels that react to specific features regardless of where they are present in an image or in a chunk of text. For example, the 2 images below are only slightly different for a CNN, since they are made of basically the same features. When analyzing text of a language where words position is a heavy grammatical constrain transformer might perform better, whereas for languages with a very relaxed positional grammar, CNN might be a better option.

$\endgroup$
1
  • $\begingroup$ It seems that you assume that a CNN can also perform a 1d convolution and I think the OP assumes that CNNs perform only 2d convolutions, which are applicable to 2d data (like images), and that's where maybe the doubt lies. So, should probably edit your answer to address this issue of different convolutions (1d, 2d, 3d). $\endgroup$
    – nbro
    May 16 at 8:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .