1
$\begingroup$

I'm trying to distinguish between the fundamental structures of the convolutional neural network and the recurrent neural network. Convolutional neural networks build a hierarchical model from the data using the convolutional layers. Recurrent neural networks build a sequential model from the data using sequential passing of information (are "recurrent layers" (to contrast with the aforementioned convolutional layers) even a thing? This website seems to have a tag for recurrent-layers, so I guess so?). The differences here aren't clear to me; specifically, it isn't clear to me how one is "hierarchical" whereas the other is "sequential", precisely how the "hierarchical" structure differs from the "sequential" structures, and how this "hierarchical" vs "sequential" distinction / inductive bias affects the suitability (effectiveness) of the models for different tasks (images vs language is the obvious case, but I'm also interested in this in a more general/abstract sense).

P.S. With regards to the "recurrent layers" question, I'm familiar with RNN architecture and memory cells (LSTM, GRU), but I've never seen the "recurrent layer" terminology, and it isn't clear to me how any part of the RNN architecture is a "recurrent layer", as opposed to the convolutional layers of CNNs, which is clearer to me as having distinct "layers" for distinct features (for instance, in an image).

P.P.S. Given the "sequential" structure, are recurrent neural networks not a hierarchical model? Do they not model things hierarchically? Or do they also model things hierarchically (although, it seems to me that this clearly isn't as clear as with convolutional neural networks)?

$\endgroup$

1 Answer 1

1
$\begingroup$

My answer is based on these 2 papers. https://arxiv.org/pdf/2110.13985 and https://arxiv.org/pdf/2111.00396 and here is the tutorial video https://www.youtube.com/watch?v=EvQ3ncuriCM that explains these two paeprs.

As they have shown, you can re-write the sequential operations of RNN in terms of convolutions. Therefore mathematically speaking CNNs can have an equiavalent sequential version that takes one pixel at a time and updates the hidden state and then get the next pixel and so on, until all the pixels of the image is fed to the RNN model.

$\endgroup$

You must log in to answer this question.