Why does this neural network require 2 hidden layers to completely fit my data?

Question

I have been working toward implementing my own Neural Network library in C++ for fun.

I have managed to implement an XOR solving network based on widely available information. Now, I wanted to try a classification problem and use the cross-entropy as my loss function.

I have created a dataset of 729 records (all possible combinations of 3 numbers between 0 and 8):

Input Features | expected Output
--------------------------------
{0, 0, 0}      | {0.0, 1.0}
{1, 0, 0}      | {1.0, 0.0}
...
{8, 8, 7}      | {0.0, 1.0}
{8, 8, 8}      | {0.0, 1.0}

The output should be {0.0, 1.0} when there is no 1 in the input, and {1.0, 0.0} when there is a 1 in the input. (the inputs are normalised to the -1.0 to 1.0 range).

I was having a hard time getting my network to correctly accomplish this classification. Because I had the whole dataset, I tried training it on the complete dataset in an attempt to have it fit perfectly. I wanted to see it fit perfectly to provide me some assurance that my implementation was correct.

Numerous attempts failed:

# hidden | learning |  results
  units  |  rate    | (accuracy)
--------------------------------
3        | 0.00001  | 72%
3        | 0.000001 | 60%
5        | 0.0001   | 80%
5        | 0.00001  | 75%
6        | 0.0001   | 79%
6        | 0.001    | 91%
10       | 0.001    | 72%
10       | 0.0001   | 91%
10       | 0.00001  | 71%
20       | 0.0001   | 91%
20       | 0.00001  | 74%
20       | 0.002    | 77%

These were all over 20k epochs, a momentum of 0.5, relu activation on all units, softmax and cross-entropy loss for the output. Some of these were unreliable too, meaning sometimes they would not learn at all, but starting with a new set of random weights worked.

You can see that it capped out at 91% accuracy. But, if I add a second hidden layer, so that my network looks like:

Input Layer | Hidden Layer 1 | Hidden Layer 2 | Output Layer
      3     |        6       |        2       |      2

I instantly get 100% accuracy using 0.001 learning rate after only 1000 epochs.

Why does this additional hidden layer work? Is there a way to know when I should use additional layers?

Answer 1

Your data is simply too non linear. You can check it by simply plotting the 2 different classes with different colors on a 3d plot. Unfortunately this is not always possible, and there's no rule of thumb to establish when the data are too non linear or how many layer you should use.

But in general you should be aware that every layer in a neural network works as a linear transformation of you data coordinates, with some non linearity added only by the final activation function, but in general a single layer lead to an almost linear mapping, hence the failure of all your attempts with a single layer.

This blog post offer beautiful visualizations of what I mean with an in depth analysis of the limitations of the classic neural networks design (hidden layer = $Wx + b$). It surely offer lot of insights even though I repeat again that there's no explicit answer to your question about how to calculate or predict how many layers are required for a task.

Also just a technical note, combinations implies no repetition, so your dataset is actually a triple cartesian product of all numbers between 0 and 8.

code to generate the plot (in python):

import matplotlib.pyplot as plt
import itertools
import numpy as np


def main():
    # generate prods data
    numbers = [i for i in range(0, 9, 1)]
    prods = [c for c in itertools.product(numbers, repeat=3)]
    prods = np.array(prods)
    # get labels
    labels = np.array([1 if 1 in prods[i, :] else 0 for i in range(prods.shape[0])]).T
    # separate data in classes 1 and 0
    prods_0 = prods[labels == 0]
    prods_1 = prods[labels == 1]
    # plot data
    fig = plt.figure()
    ax = plt.axes(projection="3d")
    ax.scatter3D(prods_0[:, 0], prods_0[:, 1], prods_0[:, 2])
    ax.scatter3D(prods_1[:, 0], prods_1[:, 1], prods_1[:, 2])
    plt.show()


if __name__ == "__main__":
    main()