# ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I tested my neural network with following architecture:

Input Layer: 5 nodes
Hidden Layer: 2 nodes (ReLU)
Output Layer: 1 node (Linear Combination)

Error: Squared Error


I trained the neural network for 1000 times over the same input and target output:

Input: [[1, 2, 3, 4, 5], [1, 2, 3, 4, 6]]
Target Output: [, ]


I expected the network to learn the sum function. However, the network ended up learning a constant function i.e weights were all negative for first layer and the bias values were negative numbers, thus application of ReLU function to it resulted in all 0's. Thus the output was simply the bias values for output layer which was 15.5

How should I interpret the above written results? I could think of a few reasons:

1. Should I consider that the network converged to a local optimum?
2. My test dataset (synthetic) was very poor. Had there been negative numbers, I could have ended up with better results?

I tried to verify the 2nd point but it so happened that the results became no better. I used:

Input: [[1, 2, 3, 4, 5], [-1, -2, -3, -4, -6]]
Target Output: [, [-16]]


It so happened that the neural network was able to evaluate both the training inputs accurately i.e 15 and -16. However, still it outputs 15 for case [1, 2, 3, 4, 6] instead of expected 16 as the weights for first layer are negative.

This made me believe that my training dataset is poor but then I tried training on a 1000 random test inputs, and the results were very poor. The weights became very large. I really can't understand what the problem is. I doubt that there might be some error in my implementation.

Another observation was: I initialized the weights and biases to optimal values i.e values that correspond to sum function:

 'W': [[ 1., -1.],
[ 1., -1.],
[ 1., -1.],
[ 1., -1.],
[ 1., -1.]]
'b':  [0., 0.]

'W':  [[ 1.],
[-1.]]
'b':  [0.]


I ran the training on that 1000 length training set but there was no effect on parameters as the error was in any case 0. Why wasn't my neural network able to learn these parameters.

For reference this is my code for neural network (hard coded for 3 layer network):

class NeuralNetwork:
def __init__(self, layers, alpha):
self.num_layers = len(layers) # has to be 3
self.layers = layers
self.alpha = alpha
self.weights = [{'W': None, 'b': None} for i in range(self.num_layers - 1)]
for i in range(self.num_layers - 1):
self.weights[i]['W'] = np.array([[np.random.normal(0, np.sqrt(2/layers[i])) for ii in range(layers[i+1])] for jj in range(layers[i])])
self.weights[i]['b'] = np.array([np.random.normal(0, np.sqrt(2/layers[i])) for ii in range(layers[i+1])])

def evaluate(self, input_feature):
psi = input_feature @ self.weights['W'] + self.weights['b']
x = np.maximum(psi, 0)
y = x @ self.weights['W'] + self.weights['b']
return y

def update_weights(self, training_input, target_output):
training_output = self.evaluate(training_input)

dely = target_output - training_output

db1  = np.sum(dely, axis = 0)
dw1  = np.sum(a*dely, axis = 0).T

da   = dely @ (self.weights['W'].T)
z    = training_input @ self.weights['W'] + self.weights['b']
dz   = np.maximum(z, 0) * da

db0  = np.sum(dz, axis = 0).T
dw0  = training_input.T @ dz

self.weights['W'] += self.alpha * dw0
self.weights['b'] += self.alpha * db0
self.weights['W'] += self.alpha * dw1
self.weights['b'] += self.alpha * db1