2
$\begingroup$

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training.

The initial code is without biases, which I'm trying to implement, forward and back calculations. From different internet sources I see that bias is just like other weights with a static input value 1, and backpropagation calculation should be similar and simplier.

But my current code version is not working - with the same input I get very different results from ~0.002 to ~0.99.

Please help me to fix biases calculations. Probably lines marked with ???. Here is a Python 2 testing code:

import numpy as np


# Sigmoid and it's derivative
def nonlin(x, deriv=False):
    if (deriv == True):
        return x*(1-x)

    return 1/(1+np.exp(-x))


X = np.array([[0,0,1],
              [0,1,1],
              [1,0,1],
              [1,1,1],
              [1,1,1]])

Y = np.array([[0],
              [1],
              [1],
              [0],
              [0]])

# Static initial hidd. layer weights for testing
wh = np.array([[-0.16258307,  0.43597283, -0.99471565, -0.39715906],
               [-0.70551921, -0.81601352, -0.62549935, -0.30959772],
               [-0.20477763,  0.07532473, -0.15920573,  0.3694664 ]])
# Static initial output layer weights for testing
wo = np.array([[-0.59572295],
               [ 0.74949506],
               [-0.95195878],
               [ 0.33625405]])

# Hidden layer's biases
biasH = 2 * np.random.random((1, 4)) - 1  # ???
# Output neuron's bias
biasO = 2 * np.random.random((1, 1)) - 1  # ???
# Static hidden layer's biases input
biasInputH = np.array([[1, 1, 1, 1]])     # ???
# Static output layer's bias input
biasInputO = np.array([[1]])              # ???


# Number of iterations to teach
for j in xrange(60000):

    # Feedforward
    h = nonlin(np.dot(X, wh) + biasH)
    o = nonlin(np.dot(h, wo) + biasO)

    # Calculate partial derivatives & errors
    o_error = Y - o

    if (j % 10000) == 0:
        print "Error:" + str(np.mean(np.abs(o_error)))

    o_delta =  o_error * nonlin(o,     deriv=True)
    o_biases = o_error * nonlin(biasO, deriv=True)  # ???

    h_error =  o_delta.dot(wo.T)
    h_delta =  h_error * nonlin(h,     deriv=True)
    h_biases = h_error * nonlin(biasH, deriv=True)  # ???

    # Update weights and biases
    wo += h.T.dot(o_delta)
    wh += X.T.dot(h_delta)

    # biasH += biasInputH.dot(h_delta)  # ???
    # biasO += biasInputO.dot(o_delta)  # ???


# Try new data
data = np.array([1,0,0])

print "weights 0:", wh
print "weights 1:", wo
print "biases 0:",  biasH
print "biases 1:",  biasO
print "input:   ",  data

h = nonlin(np.dot(data, wh))
print "hidden:  ", h
print "output:  ", nonlin(np.dot(h, wo))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.