# Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training.

The initial code is without biases, which I'm trying to implement, forward and back calculations. From different internet sources I see that bias is just like other weights with a static input value 1, and backpropagation calculation should be similar and simplier.

But my current code version is not working - with the same input I get very different results from ~0.002 to ~0.99.

Please help me to fix biases calculations. Probably lines marked with ???. Here is a Python 2 testing code:

import numpy as np

# Sigmoid and it's derivative
def nonlin(x, deriv=False):
if (deriv == True):
return x*(1-x)

return 1/(1+np.exp(-x))

X = np.array([[0,0,1],
[0,1,1],
[1,0,1],
[1,1,1],
[1,1,1]])

Y = np.array([,
,
,
,
])

# Static initial hidd. layer weights for testing
wh = np.array([[-0.16258307,  0.43597283, -0.99471565, -0.39715906],
[-0.70551921, -0.81601352, -0.62549935, -0.30959772],
[-0.20477763,  0.07532473, -0.15920573,  0.3694664 ]])
# Static initial output layer weights for testing
wo = np.array([[-0.59572295],
[ 0.74949506],
[-0.95195878],
[ 0.33625405]])

# Hidden layer's biases
biasH = 2 * np.random.random((1, 4)) - 1  # ???
# Output neuron's bias
biasO = 2 * np.random.random((1, 1)) - 1  # ???
# Static hidden layer's biases input
biasInputH = np.array([[1, 1, 1, 1]])     # ???
# Static output layer's bias input
biasInputO = np.array([])              # ???

# Number of iterations to teach
for j in xrange(60000):

# Feedforward
h = nonlin(np.dot(X, wh) + biasH)
o = nonlin(np.dot(h, wo) + biasO)

# Calculate partial derivatives & errors
o_error = Y - o

if (j % 10000) == 0:
print "Error:" + str(np.mean(np.abs(o_error)))

o_delta =  o_error * nonlin(o,     deriv=True)
o_biases = o_error * nonlin(biasO, deriv=True)  # ???

h_error =  o_delta.dot(wo.T)
h_delta =  h_error * nonlin(h,     deriv=True)
h_biases = h_error * nonlin(biasH, deriv=True)  # ???

# Update weights and biases
wo += h.T.dot(o_delta)
wh += X.T.dot(h_delta)

# biasH += biasInputH.dot(h_delta)  # ???
# biasO += biasInputO.dot(o_delta)  # ???

# Try new data
data = np.array([1,0,0])

print "weights 0:", wh
print "weights 1:", wo
print "biases 0:",  biasH
print "biases 1:",  biasO
print "input:   ",  data

h = nonlin(np.dot(data, wh))
print "hidden:  ", h
print "output:  ", nonlin(np.dot(h, wo))