Does a second-order fully-connected layer have any uses?

Question

I was thinking about implementing second-order regression via a fully-connected layer, and I came up with this:

• $X$ is the input data, shaped $(features, batch\_number)$.
• $w0$ is the bias, shaped $(output\_dims,)$.
• $w1$ and $w2$ are the weights, each shaped $(output\_dims, features)$.
• $Y$ is the target data, shaped $(output\_dims, batch\_number)$.
• $\hat{Y}$ is the output of the layer: $\hat{Y} = w2 \cdot X^2 + w1 \cdot X + w0$
  • $X^2$ is the element-wise square of $X$ taken over its first axis. (I.e., what you would expect intuitively.)

We will then minimize the MSE between $Y$ and $\hat{Y}$ using gradient descent.

Is this a valid implementation of second-order regression?

Is this layer used in any deep learning architectures? What is it useful for?

Is this layer (exactly) equivalent to some MLP?

Answer 1

Is this a valid implementation of second-order regression?

No, but it is not far off.

To perform a full second-order regression, you will need all terms for $x_{i,j}x_{i,k}$ where the first index is the example and the second index the feature. This includes every combination of two input variables. In your element-wise squaring you only produce terms $x_{i,j}x_{i,j}$, only using each feature squared, so are missing a lot of potential second-order terms.

Is this layer used in any deep learning architectures?

In general no. One of the goals of deep learning is to automate the discovery of useful non-linearities - including feature interaction terms - in the hidden layers. These won't be exactly the same as your idea, but can work well enough to get good performance from the neural network on a given dataset.

Pre-calculating a specific set of interactions might help with some data sets, but comes at a high cost of squaring the number of features represented in any given layer, and thus multiplying the number of weights required between two layers by the size of the input layer again. Your original idea would only double the size of the input layer, but would be less general.

What is it useful for?

It is useful in linear regression as a way to find interactions between features. Or you can think of it as a way to automatically search through a subset of possible nonlinear interactions to see if any of those have a strong correlation to the target variable.

There is a danger though that by adding a large number of terms (and associated parameters), that you may find spurious correlations and thus overfit the training data. This is more likely, the larger the number of interactions that you add to the input. Usually this kind of feature generation is paired with regularisation, to reduce the chance of overfitting.

Is this layer (exactly) equivalent to some MLP?

No, although because MLPs can approximate any function, it should be possible to train an MLP such that it performs very similarly.

It may not hurt to insert a layer like the one you describe into some relatively small layer of an MLP, and use your idea in pracice. I expect in some circumstances it may help, but it will be a little bit of an edge case, mostly not making much difference or real advantage. However, no harm in trying it on a few sample datasets. This applies to both your idea as written, using just the squared feature terms, and to the full second order expansion - both may have their niche where they could be useful.