Typical Feed Forward Neural Networks require a fixed sized input and output. So when you have variable sized input, it seems to be common practice to pad the input with zero vectors.

Why does it not seem to be common practice to have a "is_padding" attribute? That way the network can easily distinguish between padding and actual data? Especially considering input is commonly centered around 0 by subtracting the mean and using unit variance.

  • 1
    $\begingroup$ I would guess that maybe it's simply not necessary in common cases? If groups of inputs are often related to each other, such that some features being non-zero automatically implies a bunch of other features being "real" (even if they equal $0$), then a NN should be able to automatically learn that the $0$s are only padded if they are in a bunch together, all $0$? Really just a guess though. Intuitively, I do think your idea could be beneficial in some cases. $\endgroup$ – Dennis Soemers Feb 4 '19 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.