Simple three layer neural network with backpropagation is not approximating tanh function

Question

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid function also as an activation function of this neural network.

import numpy
# scipy.special for the sigmoid function expit()
import scipy.special
# library for plotting arrays
import matplotlib.pyplot
# ensure the plots are inside this notebook, not an external window
%matplotlib inline

# neural network class definition
class neuralNetwork:


    # initialise the neural network
    def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
        # set number of nodes in each input, hidden, output layer
        self.inodes = inputnodes
        self.hnodes = hiddennodes
        self.onodes = outputnodes

        # link weight matrices, wih and who
        # weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
        # w11 w21
        # w12 w22 etc 
        self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
        self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))

        # learning rate
        self.lr = learningrate

        # activation function is the sigmoid function
        self.activation_function = lambda x: scipy.special.expit(x)  

        pass


    # train the neural network
    def train(self, inputs_list, targets_list):
        # convert inputs list to 2d array
        inputs = numpy.array(inputs_list, ndmin=2).T
        targets = numpy.array(targets_list, ndmin=2).T

        # calculate signals into hidden layer
        hidden_inputs = numpy.dot(self.wih, inputs)
        # calculate the signals emerging from hidden layer
        hidden_outputs = self.activation_function(hidden_inputs)

        # calculate signals into final output layer
        final_inputs = numpy.dot(self.who, hidden_outputs)
        # calculate the signals emerging from final output layer
        final_outputs = self.activation_function(final_inputs)

        # output layer error is the (target - actual)
        output_errors = targets - final_outputs
        # hidden layer error is the output_errors, split by weights, recombined at hidden nodes
        hidden_errors = numpy.dot(self.who.T, output_errors) 

        # BACKPROPAGATION & gradient descent part, i.e updating weights first between hidden
        # layer and output layer, 
        # update the weights for the links between the hidden and output layers
        self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))

        # update the weights for the links between the input and hidden layers, second part of backpropagation.
        self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
        pass


    # query the neural network
    def query(self, inputs_list):
        # convert inputs list to 2d array
        inputs = numpy.array(inputs_list, ndmin=2).T

        # calculate signals into hidden layer
        hidden_inputs = numpy.dot(self.wih, inputs)
        # calculate the signals emerging from hidden layer
        hidden_outputs = self.activation_function(hidden_inputs)

        # calculate signals into final output layer
        final_inputs = numpy.dot(self.who, hidden_outputs)
        # calculate the signals emerging from final output layer
        final_outputs = self.activation_function(final_inputs)

        return final_outputs

Now I try to query this network, This network has three input nodes one for each x, one node for each input. This network also has 3 output nodes, so It would classify the inputs to given outputs. Where outputs are y, y = tanh(x) function.

# number of input, hidden and output nodes
input_nodes = 3
hidden_nodes = 8
output_nodes = 3
learning_rate = 0.1

# create instance of neural network
n = neuralNetwork(input_nodes,hidden_nodes,output_nodes, learning_rate)

realInputs = []
realInputs.append(1)
realInputs.append(2)
realInputs.append(3)

# for x in (-3, 3):
#     realInputs.append(x)
#     pass

expectedOutputs = []
expectedOutputs.append(numpy.tanh(1));
expectedOutputs.append(numpy.tanh(2));
expectedOutputs.append(numpy.tanh(3));

for y in expectedOutputs:
    print(y)
    pass

training_data_list = []

# epochs is the number of times the training data set is used for training
epochs = 200

for e in range(epochs):
    # go through all records in the training data set
    for record in training_data_list:
        # scale and shift the inputs
        inputs = realInputs
        targets = expectedOutputs
        n.train(inputs, targets)
        pass
    pass

n.query(realInputs)

Outputs: desired vs ones from network with same data as training data:

0.7615941559557649
0.9640275800758169
0.9950547536867305


array([[-0.21907413],
       [-0.6424568 ],
       [-0.25772344]])

My results are completely wrong. I'm a beginner with neural networks so I wanted to build neural network without frameworks like tensor flow... Could someone help me? Thank you.

Answer 1

This is because of Vanishing Gradient Problem

What is Vanishing Gradient Problem ?

when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier layers learn very slowly as compared to the neurons in the later layers in the Hierarchy. The Earlier layers in the network are slowest to train.

Reason

Sigmoid function, squishes a large input space into a small input space between 0 and 1. Therefore a large change in the input of the sigmoid function will cause a small change in the output. Hence, the derivative becomes small.

Solution:

Use Activation function as ReLu

Reference:

Vanishing Gradient Solution