When is it necessary to manually extract features to feed into the neural network rather than providing raw data?

Question

Usually, Neural Networks uses raw data. You do not need to extract features manually. NN's can find & extract good features which is a pattern of an image, signal or any kind of data. When we check layer outputs in a NN, we can see and visualize how NNs extract features.

Do neural networks extract features by themselves every time? When is it necessary to manually extract or engineer features to feed into the neural network rather than providing raw data?

For example, I had a time series sensor data. When I use LSTM & GRU on a raw dataset, I had bad test accuracy but when I extract some features manually I had really good test set accuracy results. I extract Fast Fourier Transform, Cross-correlation features which helped a lot to increase accuracy. "Extraction of features manually" helped to solve my problem.

Answer 1

Yes, neural networks learn features themselves freeing you from the need to manually engineer them. I will illustrate it here with a toy problem.

Let's assume that we want to learn the areas of parallelograms built on pairs of vectors:

The input data are six coordinates: $(x_1, y_1, x_2, y_2, x_3, y_3)$.

import numpy as np

n_tr = 1000 # training data
x_tr = np.random.uniform(low=-1.0, high=1.0, size=(n_tr, 6))

n_ts = 100 # test data                                        
x_ts = np.random.uniform(low=-1.0, high=1.0, size=(n_ts, 6)) 

The targets (areas) are $y = |ad-bc|$, where $a=x_3-x_1$, $b=y_3-y_1$, $c=x_2-x_1$, $d=y_2-y_1$.

a_tr = x_tr[:,4] - x_tr[:,0] # x_3 - x_1
b_tr = x_tr[:,5] - x_tr[:,1] # y_3 - y_1
c_tr = x_tr[:,2] - x_tr[:,0] # x_2 - x_1
d_tr = x_tr[:,3] - x_tr[:,1] # y_2 - y_1
y_tr = np.abs(a_tr*d_tr-b_tr*c_tr)

a_ts = x_ts[:,4] - x_ts[:,0] # x_3 - x_1
b_ts = x_ts[:,5] - x_ts[:,1] # y_3 - y_1
c_ts = x_ts[:,2] - x_ts[:,0] # x_2 - x_1
d_ts = x_ts[:,3] - x_ts[:,1] # y_2 - y_1
y_ts = np.abs(a_ts*d_ts-b_ts*c_ts)

To learn the areas from coordinates, I will use my favorite machine learning library super_magic_learn

from super_magic_learn import wonder_network

wonder_network.init()

It will initialize a network with random activation functions in neurons, and random connections between them having random weights. It also randomly assigns some neurons as inputs, while other outputs or internal ones.

Then I train my network

wonder_network.fit(x_tr, y_tr, use_wand=True)

During training, the activation functions inside neurons change, the connections between neurons form, disappear, and form again, and their weights are adjusted. Some neurons organize in layers, the number of neurons in each layer changes, and finally the trained network is as follows:

It solves the task with 100% accuracy for both the training and test data, and it solves it using only raw data: coordinates. No need to engineer features.

However, you probably don't have access to the library super_magic_learn. Let's see what can we do with a slightly more inferior tensorflow

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential(
    [
        layers.Dense(64, activation="tanh", input_dim=6),
        layers.Dense(4, activation="tanh"),
        layers.Dense(1),
    ]
)

optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)

model.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_squared_error'])

model.fit(x_tr, y_tr, epochs=512, batch_size=64, validation_split = 0.2, verbose=1)
y_pr = model.predict(x_ts)

Now calculate the performance on the test set

from sklearn.metrics import r2_score
print('Rsq:', r2_score(y_ts,y_pr))

$R^2$:

Rsq: 0.4802872880495598

Not good. What will happen, if I engineer some features?

Let's train the same model but instead of feeding it with raw data, the inputs will be the following manually engineered features: $a=x_3-x_1$, $b=y_3-y_1$, $c=x_2-x_1$, $d=y_2-y_1$ (don't forget to change input_dim=4 in the first layer).

x_tr = np.c_[a_tr, b_tr, c_tr, d_tr]
x_ts = np.c_[a_ts, b_ts, c_ts, d_ts]

$R^2$:

Rsq: 0.8841499533897564

Now it is much better. Less than 100% though.

Why neural network in tensorflow performs poorly on raw data and needs feature engineering while the super_magic_learn works perfectly on raw data and does not need any feature engineering?

The reason is that tensorflow or any other library that I know, is much more restricted than my beloved super_magic_learn. The restrictions are as follows (note a very small problem though: super_magic_learn does not exist but I wish it were):

• You have only a very small number of activation functions to choose from, like tanh, relu and a handful of others.
• The activation functions stay fixed during training. You cannot change them.
• You cannot add/remove layers.
• You cannot change the number of neurons in the layers.
• You cannot add/remove connections between the neurons.
• You have to organize your neurons only in layers, no other arrangement is allowed.
• During training, the network cannot learn the most suitable architecture for the task. E.g., it cannot reorganize itself taking into account the symmetries of the problem.
• etc ...
• Basically, the only thing you can do during training is to learn the weights.

The textbooks are right: ideally, a neural network should learn just from the raw data. But this is true only about my idealized library and not so much about existing real-world implementations.

To make a network really learn features for any task, it should be freed from these restrictions.

If you put so many restrictions on the architecture, activation functions, and other parameters, so that they cannot be learned from the data during training, then you have to engineer them yourself and adjust them manually for your task. If you engineer them correctly then your network will learn happily from the raw data. But it might perform poorly on other tasks.

Such is the case with convolutional neural networks. They were designed taking into account transnational equivariance of features in images that's why they can learn features from raw image data. However, they don't necessarily perform well in other domains.

Answer 2

Feature engineering may be necessary when one cannot achieve acceptable error rate — within a budget or in principle.

NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That's why starting fat and pruning later deems superior, although not always affordable.)

Not a necessity, but a no-brainer nonetheless: having strong insight into underlying processes is a clear call for feature engineering. Let's say, it's apparent that known meaningful transformation can hardly be approximated by (reasonably small) subnetwork: mapping between periodic and non-periodic functions, as an example.

Personally in my check-list, feature stage is coupled with a reminder to search for a domain expert. In some practical sense features are being extracted from people rather than data. Our neural networks are pretty good too!