# Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered:

My first attempt was using tf.nn.relu activation function, but output was stuck with whatever number of training steps. So I guessed it could be because of dead ReLU units, thus I changed the activation function in hidden layers to tf.nn.leaky_relu, but it's still no good.

It works when all hidden layers come with tf.sigmoid, yes, but why doesn't ReLU work here? dead ReLU units, or exploding gradients, or anything else?

Source code (TensorFlow):

#core
import time;

#libs
import tensorflow        as tf;
import matplotlib.pyplot as pyplot;

#mockup to emphasize value name
def units(Num):
return Num;
#end def

#PROGRAMME ENTRY POINT==========================================================
#data
#https://i.imgur.com/uVOxZR7.png
X = [[1,1],[1,2],[1,3],[2,1],[2,2],[2,3],[3,1],[3,2],[3,3],[4,1],[4,2],[4,3],[5,1],[6,1]];
Y = [[0],  [1],  [0],  [1],  [0],  [1],  [0],  [2],  [1],  [1],  [1],  [0],  [0],  [1]  ];
Max_X      = 6;
Max_Y      = 2;
Batch_Size = 14;

#normalise
for I in range(len(X)):
X[I][0] /= Max_X;
X[I][1] /= Max_X;
Y[I][0] /= Max_Y;
#end for

#model
Input     = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,2]);
Expected  = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,1]);

#RELU DOESN'T WORK, DEAD RELU? SIGMOID WORKS BUT SLOW.
#CHANGE TO tf.sigmoid OR tf.tanh AND IT WORKS:
activation_fn = tf.nn.leaky_relu;

#1
Weight1   = tf.Variable(tf.random_uniform(shape=[2,units(60)], minval=-1, maxval=1));
Bias1     = tf.Variable(tf.random_uniform(shape=[  units(60)], minval=-1, maxval=1));
Hidden1   = activation_fn(tf.matmul(Input,Weight1) + Bias1);

#2
Weight2   = tf.Variable(tf.random_uniform(shape=[60,units(50)], minval=-1, maxval=1));
Bias2     = tf.Variable(tf.random_uniform(shape=[   units(50)], minval=-1, maxval=1));
Hidden2   = activation_fn(tf.matmul(Hidden1,Weight2) + Bias2);

#3
Weight3   = tf.Variable(tf.random_uniform(shape=[50,units(40)], minval=-1, maxval=1));
Bias3     = tf.Variable(tf.random_uniform(shape=[   units(40)], minval=-1, maxval=1));
Hidden3   = activation_fn(tf.matmul(Hidden2,Weight3) + Bias3);

#4
Weight4   = tf.Variable(tf.random_uniform(shape=[40,units(30)], minval=-1, maxval=1));
Bias4     = tf.Variable(tf.random_uniform(shape=[   units(30)], minval=-1, maxval=1));
Hidden4   = activation_fn(tf.matmul(Hidden3,Weight4) + Bias4);

#5
Weight5   = tf.Variable(tf.random_uniform(shape=[30,units(20)], minval=-1, maxval=1));
Bias5     = tf.Variable(tf.random_uniform(shape=[   units(20)], minval=-1, maxval=1));
Hidden5   = activation_fn(tf.matmul(Hidden4,Weight5) + Bias5);

#out
Weight6   = tf.Variable(tf.random_uniform(shape=[20,units(1)], minval=-1, maxval=1));
Bias6     = tf.Variable(tf.random_uniform(shape=[   units(1)], minval=-1, maxval=1));
Output    = tf.sigmoid(tf.matmul(Hidden5,Weight6) + Bias6);

Loss      = tf.reduce_sum(tf.square(Expected-Output));
Training  = Optimiser.minimize(Loss);

#training
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed   = {Input:X, Expected:Y};
Losses = [];
Start  = time.time();

for I in range(10000):
if (I%1000==0):
Lossvalue = Sess.run(Loss, feed_dict=Feed);
Losses   += [Lossvalue];

if (I==0):
print("Loss:",Lossvalue,"(first)");
else:
print("Loss:",Lossvalue);
#end if

Sess.run(Training, feed_dict=Feed);
#end for

Lastloss = Sess.run(Loss, feed_dict=Feed);
Losses  += [Lastloss];
print("Loss:",Lastloss,"(last)");

Finish = time.time();
print("Time:",Finish-Start,"seconds");

#eval
print("\nEval:");
Evalresults = Sess.run(Output,feed_dict=Feed).tolist();
for I in range(len(Evalresults)):
Evalresults[I] = [round(Evalresults[I][0]*Max_Y)];
#end for
print(Evalresults);
Sess.close();

#result: diagram
print("\nLoss curve:");
pyplot.plot(Losses,"-bo");
#eof

• There are a number of ways to check what's happening like increasing the number of nodes and hidden layers which can give you the idea whether dead ReLu is really the problem.
– user9947
Sep 12, 2019 at 11:00

I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing.

The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and architecture, and that sigmoid is more resilient to it.

I'd suggest the following changes:

• Dropping the learning rate, try 0.01 or even 0.001

• Normalising the two input features. NNs like to work with data that is mean 0, standard deviation 1, although there is some flexibility here, your values ranging from 0 to 6 are probably starting to cause minor problems

• Look at standard initialisation routines for weights, available within TensorFlow framework, such as Glorot uniform. Your random -1 to +1 is probably too high a range for the given network, and NNs - especially "deep" NNs with 3+ hidden layers - are very sensitive to how initial weights are set.

• Simplify the network architecture a little. Five hidden layers and 200 neurons seems a bit much for your goal of over-fitting to this small data set. Try something like 3 hidden layers and 50 neurons.

• Your output layer and loss function are designed for a regression task, but you mention that the goal is to identify classes. You need a softmax layer and multiclass log loss for predicting exclusive classes.

• If you have any questions about details of any of the suggestions, then please ask a new, more specific question on AI Stack Exchange or Data Science Stack Exchange. If the question is more about theory/maths then probably better here. If it's about your data or implementations using TensorFlow then Data Science would be better. Sep 12, 2019 at 11:21
• tks, i will try the suggested changes
– Dee
Sep 12, 2019 at 11:29
• i reduced number of layers, and relu now works! leaky relu too :) tks
– Dee
Sep 13, 2019 at 10:26