I was following Daniel Shiffman's tutorials on how to write your own neural network from scratch. I specifically looked into his videos and the code he provided in here. I rewrote his code in Python, however, 3 out of 4 of my outputs are the same. The neural network has two input nodes, one hidden layer with two nodes and one output node. Can anyone help me to find my mistake? Here is my full code.
import random
nn = NeuralNetwork(2,2,1)
inputs = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
targets = np.array([[0], [1], [1], [0]])
zipped = zip(inputs, targets)
list_zipped = list(zipped)
for _ in range(9000):
x, y = random.choice(list_zipped)
nn.train(x, y)
output = [nn.feedforward(i) for i in inputs]
for i in output:
print("Output ", i)
#Output [ 0.1229546] when it should be around 0
#Output [ 0.6519492] ~1
#Output [ 0.65180228] ~1
#Output [ 0.66269853] ~0
EDIT_1: I tried debugging my code by choosing all weights and bias' values to 0.5. I did this in both my code and Daniel's. This obviously ended up showing me all outputs with the same value.
After that I increased my weights and bias' values variety from [0 , 1) to [-1, 1). By running this a few times, I would sometimes get the correct output:
[ 0.93749991] # should be ~1
[ 0.93314793] # ~1
[ 0.07001175] # ~0
[ 0.06576194] # ~0
If I ran nn.train() 100 000 times, I get the correct output 2/3 times. Is this the issue of gradient descent, where it converges to the local minima?