[TL;DR]
I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons. The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White. This is the final image I get:
At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).
[Notebook Style]
I generated two classes Red and Blue on a 2D space.
Red are points on Unit Circle
size_ = 200
classA_r = np.random.uniform(low = 0, high = 1, size = size_)
classA_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
classA_x = classA_r * np.cos(classA_theta)
classA_y = classA_r * np.sin(classA_theta)
and Blue are points on a Circle Ring with radius limits (3,4).
classB_r = np.random.uniform(low = 2, high = 3, size = size_)
classB_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
classB_x = classB_r * np.cos(classB_theta)
classB_y = classB_r * np.sin(classB_theta)
I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons.
hidden_layers = 15
inputs = Input(shape=(2,))
dnn = inputs
for l_no in range(hidden_layers):
dnn = Dense(2, activation='tanh', name = "layer_{}".format(l_no))(dnn)
outputs = Dense(2, activation='softmax', name = "layer_out")(dnn)
model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy'])
The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White.
limit = 4
step = 0.2
grid = []
x = -limit
while x <= limit:
y = -limit
while y <= limit:
grid.append([x, y])
y += step
x += step
grid = np.array(grid)
prediction = model.predict(grid)
This is the final image I get:
xs = []
ys = []
cs = []
for point in grid:
xs.append(point[0])
ys.append(point[1])
for pred in prediction:
cs.append(pred[0])
plt.scatter(xs, ys, c = cs, s=70, cmap = 'gray')
plt.scatter(classA_x, classA_y, c = 'r', s= 50)
plt.scatter(classB_x, classB_y, c = 'b', s= 50)
plt.show()
At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).
What I mean by a closed shape, take a look at the second image which was generated by using 3 neurons at each layer:
for l_no in range(hidden_layers):
dnn = Dense(3, activation='tanh', name = "layer_{}".format(l_no))(dnn)
[According to Marked Answer]
from keras import backend as K
def x_squared(x):
x = K.abs(x) * K.abs(x)
return x
hidden_layers = 3
inputs = Input(shape=(2,))
dnn = inputs
for l_no in range(hidden_layers):
dnn = Dense(2, activation=x_squared, name = "layer_{}".format(l_no))(dnn)
outputs = Dense(2, activation='softsign', name = "layer_out")(dnn)
model.compile(optimizer='adam',
loss='mean_squared_error',
metrics=['accuracy'])
I get: