# Neural Net not training biases correctly

This is an XOR neural net I built in Python. When I run it with random weights and keep the biases constant, it trains them perfectly and reaches the global minima, but when i add the biases in so that it has to train them too, it doesn't work at all. I think its getting stuck in a local minima because of my training function but i don't know for sure. Attached is the code to only train the weights.

#!/usr/bin/env python
import math
import random

class neuralNetwork:
def __init__(self):
self.input = [[0,0],[1,0],[1,1],[0,1]] #four different sets of two inputs
self.expected = [1,1,0,1],[0,1,1,1],[0,1,0,1] #Optimal outputs for all three nodes
self.weights = [random.randrange(-10,10),random.randrange(-10,10)],[random.randrange(-10,10),random.randrange(-10,10)],[random.randrange(-10,10),random.randrange(-10,10)]
#self.weights = [-20,-20,20,20,20,20] #Optimal Weights
self.bias = [30,-10,-30] #one bias for each node
#self.bias = [30,-10,-30] #Optimal biases
self.out = [0,0,0,0],[0,0,0,0],[0,0,0,0] #Actual output for every node
self.rmse= [0,0,0]
self.error = [0,0,0,0],[0,0,0,0],[0,0,0,0]
self.learnRate = 1

def nandGate(self):
for i in range(0,4):
self.out[i] = self.input[i]*self.weights+self.input[i]*self.weights+self.bias
self.out[i] = 1 / (1 + math.exp(-self.out[i]))

def orGate(self):
for i in range(0,4):
self.out[i] = self.input[i]*self.weights+self.input[i]*self.weights+self.bias
self.out[i] = 1 / (1 + math.exp(-self.out[i]))

def andGate(self):
for i in range(0,4):
self.out[i] = self.out[i]*self.weights+self.out[i]*self.weights+self.bias
self.out[i] = 1 / (1 + math.exp(-self.out[i]))

def calcrmse(self):
for p in range(0,3):
self.rmse[p] = 0
for p in range(0,3):
for i in range(0,4):
self.error[p][i] = self.expected[p][i] - self.out[p][i]
self.rmse[p] += self.error[p][i]

def train(self):
for q in range(0,1000000):
self.nandGate()
self.orGate()
self.andGate()
self.calcrmse()
for i in range(0,3):
#self.bias[i] += self.learnRate*self.rmse[i]
for p in range(0,2):
self.weights[i][p] += self.learnRate*self.rmse[i]
#print("Iter: "+str(q))
#print("WEIGHTS: "+str(self.weights))
#print("BIAS: "+str(self.bias))
#print("NAND OUT: "+str(self.out))
#print("OR OUT: "+str(self.out))
print("AND OUT: "+str(self.out))
#print

x = neuralNetwork()
x.train()

• I'm voting to close this question as off-topic because it's right fit in cross validated community or stack-overflow,given by what the scope of this community says. – quintumnia Jun 23 '18 at 16:58

Your code has a lot of errors and may surely get this question down-voted but anyways assuming its correct there are a few things which i noticed in your code:

self.weights = [random.randrange(-10,10),random.randrange(-10,10)],[random.randrange(-10,10),random.randrange(-10,10)],[random.randrange(-10,10),random.randrange(-10,10)]

This creates a tuple (Python 2.7), and tuples cannot be modified in Python.

Secondly, defining learning rate as 1 is never a good idea, it might cause your predictor to bounce and gain exponential values of weights (totally unpredictable). Instead you can reduce the learning rate and keep the number of epochs high.

self.out[i] = self.input[i]*self.weights+self.input[i]*self.weights+self.bias
self.out[i] = 1 / (1 + math.exp(-self.out[i]))


I have no idea why are you defining your output like this. Gates have binary output so where is your final classifier that decides whether output is 0 or 1?

for i in range(0,3):
#self.bias[i] += self.learnRate*self.rmse[i]
for p in range(0,2):
self.weights[i][p] += self.learnRate*self.rmse[i]


In this piece, where is the input vector? Training weights always require the input vector for which you are training. You are implementing batch learning but you are implementing it quite incorrectly. The formula for batch learning is E*x where x is the input vector and E is the error. What you have done is just summed up all the errors and used no input vector also.

You ultimately print self.out() but what exactly does it signify? All values will be positive in that list.

I think you have confused perceptron learning with delta rule and batch learning with online learning. If you are getting correct answers it maybe due to some fluke. You should write the code again and try to see what exactly is it that you are doing as your code is too garbled to know what exactly is going on.