1
$\begingroup$

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid function also as an activation function of this neural network.

import numpy
# scipy.special for the sigmoid function expit()
import scipy.special
# library for plotting arrays
import matplotlib.pyplot
# ensure the plots are inside this notebook, not an external window
%matplotlib inline

# neural network class definition
class neuralNetwork:


    # initialise the neural network
    def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
        # set number of nodes in each input, hidden, output layer
        self.inodes = inputnodes
        self.hnodes = hiddennodes
        self.onodes = outputnodes

        # link weight matrices, wih and who
        # weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
        # w11 w21
        # w12 w22 etc 
        self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
        self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))

        # learning rate
        self.lr = learningrate

        # activation function is the sigmoid function
        self.activation_function = lambda x: scipy.special.expit(x)  

        pass


    # train the neural network
    def train(self, inputs_list, targets_list):
        # convert inputs list to 2d array
        inputs = numpy.array(inputs_list, ndmin=2).T
        targets = numpy.array(targets_list, ndmin=2).T

        # calculate signals into hidden layer
        hidden_inputs = numpy.dot(self.wih, inputs)
        # calculate the signals emerging from hidden layer
        hidden_outputs = self.activation_function(hidden_inputs)

        # calculate signals into final output layer
        final_inputs = numpy.dot(self.who, hidden_outputs)
        # calculate the signals emerging from final output layer
        final_outputs = self.activation_function(final_inputs)

        # output layer error is the (target - actual)
        output_errors = targets - final_outputs
        # hidden layer error is the output_errors, split by weights, recombined at hidden nodes
        hidden_errors = numpy.dot(self.who.T, output_errors) 

        # BACKPROPAGATION & gradient descent part, i.e updating weights first between hidden
        # layer and output layer, 
        # update the weights for the links between the hidden and output layers
        self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))

        # update the weights for the links between the input and hidden layers, second part of backpropagation.
        self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
        pass


    # query the neural network
    def query(self, inputs_list):
        # convert inputs list to 2d array
        inputs = numpy.array(inputs_list, ndmin=2).T

        # calculate signals into hidden layer
        hidden_inputs = numpy.dot(self.wih, inputs)
        # calculate the signals emerging from hidden layer
        hidden_outputs = self.activation_function(hidden_inputs)

        # calculate signals into final output layer
        final_inputs = numpy.dot(self.who, hidden_outputs)
        # calculate the signals emerging from final output layer
        final_outputs = self.activation_function(final_inputs)

        return final_outputs

Now I try to query this network, This network has three input nodes one for each x, one node for each input. This network also has 3 output nodes, so It would classify the inputs to given outputs. Where outputs are y, y = tanh(x) function.

# number of input, hidden and output nodes
input_nodes = 3
hidden_nodes = 8
output_nodes = 3
learning_rate = 0.1

# create instance of neural network
n = neuralNetwork(input_nodes,hidden_nodes,output_nodes, learning_rate)

realInputs = []
realInputs.append(1)
realInputs.append(2)
realInputs.append(3)

# for x in (-3, 3):
#     realInputs.append(x)
#     pass

expectedOutputs = []
expectedOutputs.append(numpy.tanh(1));
expectedOutputs.append(numpy.tanh(2));
expectedOutputs.append(numpy.tanh(3));

for y in expectedOutputs:
    print(y)
    pass

training_data_list = []

# epochs is the number of times the training data set is used for training
epochs = 200

for e in range(epochs):
    # go through all records in the training data set
    for record in training_data_list:
        # scale and shift the inputs
        inputs = realInputs
        targets = expectedOutputs
        n.train(inputs, targets)
        pass
    pass

n.query(realInputs)

Outputs: desired vs ones from network with same data as training data:

0.7615941559557649
0.9640275800758169
0.9950547536867305


array([[-0.21907413],
       [-0.6424568 ],
       [-0.25772344]])

My results are completely wrong. I'm a beginner with neural networks so I wanted to build neural network without frameworks like tensor flow... Could someone help me? Thank you.

$\endgroup$
  • $\begingroup$ Hi and welcome to AI SE! It may be a good idea if you describe a little bit your code. For example, 1. Where are you implementing the back-propagation algorithm? 2. Where are you implementing the gradient descent step? 3. Why do you have 3 input nodes? 4. Why do you have 3 output nodes? $\endgroup$ – nbro Mar 24 at 0:19
  • $\begingroup$ @nbro Updated, added further description. $\endgroup$ – Patrick Mar 24 at 10:35
0
$\begingroup$

This is because of Vanishing Gradient Problem

What is Vanishing Gradient Problem ?

when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier layers learn very slowly as compared to the neurons in the later layers in the Hierarchy. The Earlier layers in the network are slowest to train.

Reason

Sigmoid function, squishes a large input space into a small input space between 0 and 1. Therefore a large change in the input of the sigmoid function will cause a small change in the output. Hence, the derivative becomes small. Sigmoid And its Derivative Function

Solution:

Use Activation function as ReLu

ReLu

Reference:

Vanishing Gradient Solution

| improve this answer | |
$\endgroup$
  • $\begingroup$ How do you know it's because of the vanishing gradient problem? OP's neural network is quite small, so, even though the OP using the sigmoid, I doubt that their problem is due to the vanishing gradient. In fact, the OP gets reasonable numbers (i.e. not too small), they are just wrong. I encourage you to have a look at their code first, before saying it's the vanishing gradient problem. $\endgroup$ – nbro Mar 24 at 1:31
  • $\begingroup$ Sure i will look into it @nbro $\endgroup$ – D_Raja Mar 24 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.