Why MLP cannot approximate a closed shape function?

Question

[TL;DR]

I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons. The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White. This is the final image I get:

At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).

[Notebook Style]

I generated two classes Red and Blue on a 2D space.
Red are points on Unit Circle

size_ = 200
classA_r = np.random.uniform(low = 0, high = 1, size = size_)
classA_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
classA_x = classA_r * np.cos(classA_theta)
classA_y = classA_r * np.sin(classA_theta)

and Blue are points on a Circle Ring with radius limits (3,4).

classB_r = np.random.uniform(low = 2, high = 3, size = size_)
classB_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
classB_x = classB_r * np.cos(classB_theta)
classB_y = classB_r * np.sin(classB_theta)

I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons.

hidden_layers = 15
inputs = Input(shape=(2,))
dnn = inputs
for l_no in range(hidden_layers):
    dnn = Dense(2, activation='tanh', name = "layer_{}".format(l_no))(dnn)
outputs = Dense(2, activation='softmax', name = "layer_out")(dnn)

model = Model(inputs=inputs, outputs=outputs)

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy'])

The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White.

limit = 4
step = 0.2
grid = []
x = -limit
while x <= limit:
    y = -limit
    while y <= limit:
        grid.append([x, y])
        y += step
    x += step
grid = np.array(grid)

prediction = model.predict(grid)

This is the final image I get:

xs = []
ys = []
cs = []
for point in grid:
    xs.append(point[0])
    ys.append(point[1])
for pred in prediction:
    cs.append(pred[0])

plt.scatter(xs, ys, c = cs, s=70, cmap = 'gray')
plt.scatter(classA_x, classA_y, c = 'r', s= 50)
plt.scatter(classB_x, classB_y, c = 'b', s= 50)
plt.show()

At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).

What I mean by a closed shape, take a look at the second image which was generated by using 3 neurons at each layer:

for l_no in range(hidden_layers):
    dnn = Dense(3, activation='tanh', name = "layer_{}".format(l_no))(dnn)

[According to Marked Answer]

from keras import backend as K
def x_squared(x):
    x = K.abs(x) * K.abs(x)
    return x
hidden_layers = 3
inputs = Input(shape=(2,))
dnn = inputs
for l_no in range(hidden_layers):
    dnn = Dense(2, activation=x_squared, name = "layer_{}".format(l_no))(dnn)
outputs = Dense(2, activation='softsign', name = "layer_out")(dnn)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])

I get:

Answer 1

In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, tanh or hyperbolic tangent).

Assuming at least one hidden layer, then the universal approximation theorem applies. How closely you can approximate any given function is limited by the number of neurons, and not strongly by the choice of activation function. However, the choice of activation function is still relevant to how good an approximation is. When you get to low numbers of neurons in a hidden layer, then the approximations are more strongly tied to the nature of the activation function.

With one neuron in a hidden layer, you can only approximate some affine transformation of the activation function. Other low numbers, such as 2, 3 etc, will still show strong tendencies for certain families of shapes. This is very similar conceptually to using limited number of frequencies in a Fourier transform - if you limit yourself to only $a_1 \text{sin}(x) + a_2 \text{sin}(2x)$ to approximate a function, then you will definitely notice the sinusoidal building blocks in any output.

I suspect that if you changed the activation function in the first hidden layer to $f(x) = x^2$ then you could get a good result with two neurons per layer. If you then took that network and tried to train it on a simple linear split, it would fail, always producing some curved closed surface that covered the training examples as best that it could - kind of the opposite problem as you are seeing with your circular pattern fitted to NN with tanh activations throughout.

One interesting thing about using $f(x) = x^2$ is that this is a deliberate choice (given knowledge about how you constructed the example) to map your input space to a new space where examples can be linearly separated. In fact this appears to be what layers in multi-layer NNs learn - each layer incrementally and progressively maps its input space to a new space where examples can be better separated in a linear fashion.