3
$\begingroup$

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape

(m, max_seq_len)

where m is the number of examples, e.g. sentences, and max_seq_len is the maximum length that a sentence can have. Some examples have smaller lengths than this max_seq_len. A solution is to pad these sentences.

One method to pad the sentences is called "zero-padding". This means that each sequence is padded with zeros. For example, given a vocabulary where each word is related to some index number, we can represent a sentence with length 4,

I am very confused

by

[23, 455, 234, 90]

Padding it to achieve a max_seq_len=7, we obtain a sentence represented by:

[23, 455, 234, 90, 0, 0, 0]

The index 0 is not part of the vocabulary.

Another method to pad is to add a padding character, e.g. <<pad>>, in our sentence:

I am very confused <<pad>>> <<pad>> <<pad>>

to achieve the max_seq_len=7. We also add <<pad>> in our vocabulary. Let's say its index is 1000. Then the sentence is represented by

[23, 455, 234, 90, 1000, 1000, 1000]

I have seen both methods used, but why is one used over the other? Are there any advantages or disadvantages comparing zero-padding with character-padding?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ At least in Keras it is the default. Sure, you can pad_sequences and use a Masking layer with an arbitrary value. But the Embedding layer, the most frequent consumer of these encodings, provides a "mask_zero" boolean parameter. $\endgroup$
    – CorruptedHeapScapeGoat
    Feb 27, 2021 at 14:14

0

Reset to default

You must log in to answer this question.