# Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid function also as an activation function of this neural network.

import numpy
# scipy.special for the sigmoid function expit()
import scipy.special
# library for plotting arrays
import matplotlib.pyplot
# ensure the plots are inside this notebook, not an external window
%matplotlib inline

# neural network class definition
class neuralNetwork:

# initialise the neural network
def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes

# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))

# learning rate
self.lr = learningrate

# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)

pass

# train the neural network
def train(self, inputs_list, targets_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T

# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)

# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)

# output layer error is the (target - actual)
output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)

# BACKPROPAGATION & gradient descent part, i.e updating weights first between hidden
# layer and output layer,
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))

# update the weights for the links between the input and hidden layers, second part of backpropagation.
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
pass

# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T

# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)

# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)

return final_outputs


Now I try to query this network, This network has three input nodes one for each x, one node for each input. This network also has 3 output nodes, so It would classify the inputs to given outputs. Where outputs are y, y = tanh(x) function.

# number of input, hidden and output nodes
input_nodes = 3
hidden_nodes = 8
output_nodes = 3
learning_rate = 0.1

# create instance of neural network
n = neuralNetwork(input_nodes,hidden_nodes,output_nodes, learning_rate)

realInputs = []
realInputs.append(1)
realInputs.append(2)
realInputs.append(3)

# for x in (-3, 3):
#     realInputs.append(x)
#     pass

expectedOutputs = []
expectedOutputs.append(numpy.tanh(1));
expectedOutputs.append(numpy.tanh(2));
expectedOutputs.append(numpy.tanh(3));

for y in expectedOutputs:
print(y)
pass

training_data_list = []

# epochs is the number of times the training data set is used for training
epochs = 200

for e in range(epochs):
# go through all records in the training data set
for record in training_data_list:
# scale and shift the inputs
inputs = realInputs
targets = expectedOutputs
n.train(inputs, targets)
pass
pass

n.query(realInputs)


Outputs: desired vs ones from network with same data as training data:

0.7615941559557649
0.9640275800758169
0.9950547536867305

array([[-0.21907413],
[-0.6424568 ],
[-0.25772344]])


My results are completely wrong. I'm a beginner with neural networks so I wanted to build neural network without frameworks like tensor flow... Could someone help me? Thank you.

• Hi and welcome to AI SE! It may be a good idea if you describe a little bit your code. For example, 1. Where are you implementing the back-propagation algorithm? 2. Where are you implementing the gradient descent step? 3. Why do you have 3 input nodes? 4. Why do you have 3 output nodes? – nbro Mar 24 at 0:19
• @nbro Updated, added further description. – Patrick Mar 24 at 10:35

This is because of Vanishing Gradient Problem

What is Vanishing Gradient Problem ?

when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier layers learn very slowly as compared to the neurons in the later layers in the Hierarchy. The Earlier layers in the network are slowest to train.

Reason

Sigmoid function, squishes a large input space into a small input space between 0 and 1. Therefore a large change in the input of the sigmoid function will cause a small change in the output. Hence, the derivative becomes small. Solution:

Use Activation function as ReLu Reference: