Why don't neural networks project the data into higher dimensions first, then reduce the size of each layer thereafter?

Question

Background

From my understanding (and following along with this blog post), (deep) neural networks apply transformations to the data such that the data's representation to the next layer (or classification layer) becomes more separate. As such, we can then apply a simple classifier(s) to the representation to chop up the regions where the different classes exist (as shown by this blog post).

If this is true and say we have some noisy data where the classes are not easily separable, would it make sense to push the input to a higher dimension, so we can more easily separate it later in the network?

For example, I have some tabular data that is a bit noisy, say it has 50 dimensions (input size of 50). To me, it seems logical to project the data to a higher dimension, such that it makes it easier for the classifier to separate. In essence, I would project the data to say 60 dimensions (layer out dim = 60), so the network can represent the data with more dimensions, allowing us to linearly separate it. (I find this similar to how SVMs can classify the data by pushing it to a higher dimension).

Question

Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?

I learned that if we have more hidden nodes than input nodes, the network will memorize rather than generalize.

Answer 1

To better understand this you should think in terms of capacity. Capacity is a theoretical notion that shows how much information your network can model.

The capacity of a network (given sufficient training) ties in directly with the bias/variance tradeoff:

• too little capacity and your network isn't able to learn the complex relationships in the data.
• too much capacity and your network has the capability of learning the noise in the dataset, besides the useful relationships.

At some point a network reaches a point where it has a high enough capacity to memorize the whole training set!

Now, by increasing the number of hidden neurons you essentially increase the capacity of your network. If the network already has enough capacity to learn the problem, then by increasing the neurons you are giving the network the capability of overfitting more easily.

Note: all the above assume that the network is trained sufficiently (i.e. no early stopping, etc).

Answer 2

The accepted answer does not answer the question

Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?

Yes, it's true that if you increase the number of hidden neurons, you generally increase the capacity (in fact, the VC dimension of neural networks is typically expressed as a function of the number of parameters), but you're also suggesting to decrease the size afterwards.

For example, in this tutorial, they use 10 hidden neurons for the first hidden layer of an MLP, while the dataset contains only 4 features, which means that there are $4*10 = 40$ weights (aka parameters). In this other tutorial, there are less than 20 features (and only a few of them are used) and the first hidden layer has 128 neurons, which means that each feature is connected to $128$ hidden neurons, so there should be $4*128 = 512$ weights. So, MLPs can easily have more hidden neurons in the first layer and weights that connect the input(s) to the hidden neurons than the number of features.

In the case of CNNs, you may have fewer scalar parameters because of the properties of CNNs, i.e. parameter sharing. For instance, if you have $64$ filters of shape $3 \times 3$ and you want to process grayscale images of shape $32 \times 32$, then you have $3 * 3 * 64 = 576 < 1024 = 32*32$. Note that the number of parameters of the first layer does not change as a function of the input in the case of CNNs, so if that same CNN processes an image of size $5 \times 5$, then the first layer contains more scalar parameters than pixels.

So, in general, NNs can project the data to higher-dimensional space. The typical NNs that project to a lower-dimensional space are auto-encoders or data compressors, in general, that's why they are called in this way.