How is dropout applied to the embedding layer's output?

Question

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(1000, 16, input_length=20), 
    tf.keras.layers.Dropout(0.2),                           # <- How does the dropout work?
    tf.keras.layers.Conv1D(64, 5, activation='relu'),
    tf.keras.layers.MaxPooling1D(pool_size=4),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

I can understand when dropout is applied between Dense layers, which randomly drops and prevents the former layer neurons from updating parameters. I don't understand how dropout works after an embedding layer.

Let's say the output shape of the Embedding layer is (batch_size,20,16) or simply (20,16) if we ignore the batch size. How is dropout applied to the embedding layer's output?

Randomly dropout rows or columns?

Answer 1

It doesn't drops rows or columns, it acts directly on scalars. The Dropout Layer keras documentation explains it and illustrates it with an example :

The Dropout layer randomly sets input units to 0 with a frequency of rate

After an Dense Layer, the Dropout inputs are directly the outputs of the Dense layer neurons, as you said. After your embedding layer, in your case, you should have rate * (16 * input_length) = 0.2 * 20 * 16 = 64 inputs set to 0 out of the 320 scalars inputs. These 64 dropped inputs are randomly selected in the 20x16 grid. Note that the Dropout rescales the non dropped inputs by multiplicating them by a factor $\frac{1}{1-rate}$.