Consider the following paragraphs from the introduction of the chapter named Recurrent Neural Networks from the textbook titled Dive into Deep Learning

So far we encountered two types of data: tabular data and image data. For the latter we designed specialized layers to take advantage of the regularity in them. In other words, if we were to permute the pixels in an image, it would be much more difficult to reason about its content of something that would look much like the background of a test pattern in the times of analog TV.

Most importantly, so far we tacitly assumed that our data are all drawn from some distribution, and all the examples are independently and identically distributed (i.i.d.). Unfortunately, this is not true for most data. For instance, the words in this paragraph are written in sequence, and it would be quite difficult to decipher its meaning if they were permuted randomly. Likewise, image frames in a video, the audio signal in a conversation, and the browsing behavior on a website, all follow sequential order. It is thus reasonable to assume that specialized models for such data will do better at describing them.

In neural networks, we generally use words: instances, examples, data points to refer to a particular row of a dataset. In general, in the case of CNN, an instance will be an image, and in the case of RNN, an instance will be a sequence of text (maybe a sentence, paragraph, or text).

Every instance contains features: in image data, pixels are generally treated as features and in the case of text data, either characters or words are generally treated as features.

With this context, let me explain my doubt

I have an issue understanding the paragraphs. The issue is the comparison of examples/instances of image data with features of text data. Pixels can be compared to words/characters and images can be compared to sentences or words or paragraphs or text documents based on the context.

In the second paragraph, it is said that the examples (images) are i.i.d. But then the images are compared with words in a paragraph, but words are features, not examples. As the paragraph is not saying that pixels are i.i.d., how can the words in paragraphs be used for comparison?

  • $\begingroup$ I am not sure I understand your question. I understand it has to do with images, words, sentences, being iid or not, and to compare words with images, but I don't understand what the specific question is. I don't understand this "As the paragraph is not saying that pixels are i.i.d. how can the words in paragraphs be used for comparison?". I mean, why do you need both the pixels and the words to be iid to compare them? If pixels and words are features, then you can compare them, in the sense that you can say that both are features. What do you mean by "compare"? $\endgroup$
    – nbro
    Mar 9, 2022 at 9:48
  • $\begingroup$ @nbro in the second paragraph, they told that examples are iid (w.r.t image data). But they are talking about the words in paragraphs. How to reconcile? $\endgroup$
    – hanugm
    Mar 9, 2022 at 11:52
  • $\begingroup$ @nbro If they say pixels are iid then they can compare with words. But they are saying images are iid then how can they compare with words instead of paragraphs? $\endgroup$
    – hanugm
    Mar 9, 2022 at 11:55
  • $\begingroup$ It seems to me you think that words are "features" and that we can't compare words with images because images are "samples" and not features. Is this your question? I would recommend that you edit your post to remove unnecessary details and just leave specific question that highlights your doubt/problem. $\endgroup$
    – nbro
    Mar 10, 2022 at 9:15
  • 1
    $\begingroup$ Thanks. I've rewritten the title to make it clearer what the problem is. Make sure that's your question. Feel free to edit again your problem to improve it ;) $\endgroup$
    – nbro
    Mar 10, 2022 at 9:47


You must log in to answer this question.