Yes, neural networks learn features themselves freeing you from the need to manually engineer them. I will illustrate it here with a toy problem.
Let's assume that we want to learn the areas of parallelograms built on pairs of vectors:
The input data are six coordinates: $(x_1, y_1, x_2, y_2, x_3, y_3)$.
import numpy as np
n_tr = 1000 # training data
x_tr = np.random.uniform(low=-1.0, high=1.0, size=(n_tr, 6))
n_ts = 100 # test data
x_ts = np.random.uniform(low=-1.0, high=1.0, size=(n_ts, 6))
The targets (areas) are $y = |ad-bc|$, where $a=x_3-x_1$, $b=y_3-y_1$, $c=x_2-x_1$, $d=y_2-y_1$.
a_tr = x_tr[:,4] - x_tr[:,0] # x_3 - x_1
b_tr = x_tr[:,5] - x_tr[:,1] # y_3 - y_1
c_tr = x_tr[:,2] - x_tr[:,0] # x_2 - x_1
d_tr = x_tr[:,3] - x_tr[:,1] # y_2 - y_1
y_tr = np.abs(a_tr*d_tr-b_tr*c_tr)
a_ts = x_ts[:,4] - x_ts[:,0] # x_3 - x_1
b_ts = x_ts[:,5] - x_ts[:,1] # y_3 - y_1
c_ts = x_ts[:,2] - x_ts[:,0] # x_2 - x_1
d_ts = x_ts[:,3] - x_ts[:,1] # y_2 - y_1
y_ts = np.abs(a_ts*d_ts-b_ts*c_ts)
To learn the areas from coordinates, I will use my favorite machine learning library
from super_magic_learn import wonder_network
It will initialize a network with random activation functions in neurons, and random connections between them having random weights. It also randomly assigns some neurons as inputs, while other outputs or internal ones.
Then I train my network
wonder_network.fit(x_tr, y_tr, use_wand=True)
During training, the activation functions inside neurons change, the connections between neurons form, disappear, and form again, and their weights are adjusted. Some neurons organize in layers, the number of neurons in each layer changes, and finally the trained network is as follows:
It solves the task with 100% accuracy for both the training and test data, and it solves it using only raw data: coordinates. No need to engineer features.
However, you probably don't have access to the library
super_magic_learn. Let's see what can we do with a slightly more inferior
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential(
layers.Dense(64, activation="tanh", input_dim=6),
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
model.fit(x_tr, y_tr, epochs=512, batch_size=64, validation_split = 0.2, verbose=1)
y_pr = model.predict(x_ts)
Now calculate the performance on the test set
from sklearn.metrics import r2_score
Not good. What will happen, if I engineer some features?
Let's train the same model but instead of feeding it with raw data, the inputs will be the following manually engineered features: $a=x_3-x_1$, $b=y_3-y_1$, $c=x_2-x_1$, $d=y_2-y_1$ (don't forget to change
input_dim=4 in the first layer).
x_tr = np.c_[a_tr, b_tr, c_tr, d_tr]
x_ts = np.c_[a_ts, b_ts, c_ts, d_ts]
Now it is much better. Less than 100% though.
Why neural network in
tensorflow performs poorly on raw data and needs feature engineering while the
super_magic_learn works perfectly on raw data and does not need any feature engineering?
The reason is that
tensorflow or any other library that I know, is much more restricted than my beloved
super_magic_learn. The restrictions are as follows (note a very small problem though:
super_magic_learn does not exist but I wish it were):
- You have only a very small number of activation functions to choose from, like tanh, relu and a handful of others.
- The activation functions stay fixed during training. You cannot change them.
- You cannot add/remove layers.
- You cannot change the number of neurons in the layers.
- You cannot add/remove connections between the neurons.
- You have to organize your neurons only in layers, no other arrangement is allowed.
- During training, the network cannot learn the most suitable architecture for the task. E.g., it cannot reorganize itself taking into account the symmetries of the problem.
- etc ...
- Basically, the only thing you can do during training is to learn the weights.
The textbooks are right: ideally, a neural network should learn just from the raw data. But this is true only about my idealized library and not so much about existing real-world implementations.
To make a network really learn features for any task, it should be freed from these restrictions.
If you put so many restrictions on the architecture, activation functions, and other parameters, so that they cannot be learned from the data during training, then you have to engineer them yourself and adjust them manually for your task. If you engineer them correctly then your network will learn happily from the raw data. But it might perform poorly on other tasks.
Such is the case with convolutional neural networks. They were designed taking into account transnational equivariance of features in images that's why they can learn features from raw image data. However, they don't necessarily perform well in other domains.