46
$\begingroup$

As far as I can tell, neural networks have a fixed number of neurons in the input layer.

If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a network. How is the varying input size reconciled with the fixed size of the input layer of the network? In other words, how is such a network made flexible enough to deal with an input that might be anywhere from one word to multiple pages of text?

If my assumption of a fixed number of input neurons is wrong and new input neurons are added to/removed from the network to match the input size I don't see how these can ever be trained.

I give the example of NLP, but lots of problems have an inherently unpredictable input size. I'm interested in the general approach for dealing with this.

For images, it's clear you can up/downsample to a fixed size, but, for text, this seems to be an impossible approach since adding/removing text changes the meaning of the original input.

$\endgroup$
38
$\begingroup$

Three possibilities come to mind.

The easiest is the zero-padding. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts.

Recurrent NNs (RNN) are a very natural NN to choose if you have texts of varying size as input. You input words as word vectors (or embeddings) just one after another and the internal state of the RNN is supposed to encode the meaning of the full string of words. This is one of the earlier papers.

Another possibility is using recursive NNs. This is basically a form of preprocessing in which a text is recursively reduced to a smaller number of word vectors until only one is left - your input, which is supposed to encode the whole text. This makes a lot of sense from a linguistic point of view if your input consists of sentences (which can vary a lot in size), because sentences are structured recursively. For example, the word vector for "the man", should be similar to the word vector for "the man who mistook his wife for a hat", because noun phrases act like nouns, etc. Often, you can use linguistic information to guide your recursion on the sentence. If you want to go way beyond the Wikipedia article, this is probably a good start.

$\endgroup$
  • 1
    $\begingroup$ What is the difference between what you called "recursive NNs", and RNNs? R is for recursive... Also, a few citations and pointers would be useful. $\endgroup$ – Eric Platon Sep 24 '16 at 2:13
  • 3
    $\begingroup$ R is for recurrent. Recurrent means adding input linearly to the same NN again and again. Recursive means inputting the data along a tree structure. $\endgroup$ – BlindKungFuMaster Sep 25 '16 at 9:29
  • $\begingroup$ @BlindKungFuMaster 0 padding with mask for MLPs is it good and does not affect badly the accuracy ? $\endgroup$ – DINA TAKLIT Feb 25 at 8:13
12
$\begingroup$

Others already mentioned:

  • zero padding
  • RNN
  • recursive NN

so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an excellent book which backs up this approach:

Consider a collection of images, where each image has a different width and height. It is unclear how to model such inputs with a weight matrix of fixed size. Convolution is straightforward to apply; the kernel is simply applied a different number of times depending on the size of the input, and the output of the convolution operation scales accordingly.

Taken from page 354, 9.7 Data Types, 3rd paragraph. You can read it further to see some other approaches.

$\endgroup$
  • 1
    $\begingroup$ It occurs to me that this approach will only work if 1) the aspect ratio (AR) of all the input images is the same, 2) you rescale all images to a given AR, or 3) you zero-pad images to force a given AR. $\endgroup$ – Matt Wenham Nov 1 '18 at 11:34
  • $\begingroup$ @Salvador Dali can "0" padding with mask be a good solution for MLPs? $\endgroup$ – DINA TAKLIT Feb 25 at 8:15
  • $\begingroup$ I can't find the text you cite in your answer in the book. Maybe it was present originally and has been removed later? Page 354 currently has a paragraph which starts similarly, but never uses variable number of layers (but variable output size instead). $\endgroup$ – jochen Apr 21 at 20:39
7
$\begingroup$

In NLP you have an inherent ordering of the inputs so RNNs are a natural choice.

For variable sized inputs where there is no particular ordering among the inputs, one can design networks which:

  1. use a repetition of the same subnetwork for each of the groups of inputs (i.e. with shared weights). This repeated subnetwork learns a representation of the (groups of) inputs.
  2. use an operation on the representation of the inputs which has the same symmetry as the inputs. For order invariant data, averaging the representations from the input networks is a possible choice.
  3. use an output network to minimize the loss function at the output based on the combination of the representations of the input.

The structure looks as follows:

network structure

Similar networks have been used to learn the relations between objects (arxiv:1702.05068).

A simple example of how to learning the sample variance of a variable sized set of values is given here (disclaimer: I'm the author of the linked article).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.