I am trying to implement a Neural Network for binary classification using python and numpy only.

My network structure is as follows:
input features: 2 [1X2] matrix
Hidden layer1: 5 neurons [2X5] matrix
Hidden layer2: 5 neurons [5X5] matrix
Output layer: 1 neuron [5X1]matrix
I have used the sigmoid activation function in all the layers.

Now lets say I use binary cross entropy as my loss function. How do I do the back propagation on these matrices to update weights?

class Layer():
    def __init__(self,number_of_neurons,number_of_inputs):
        self.weights=np.random.rand(number_of_neurons, number_of_inputs)

class NeralNetwork():
        def __init__(self, layer1, layer2,layer3):
            self.layer1 = layer1
            self.layer2 = layer2
            self.layer3 = layer3

        def sigmoid(self,x):
            return 1 / (1 + np.exp(-x))

        def derivative_sigmoid(self,x):
            return x*(1-x)

        def get_cost_value(self,Y_hat, Y):
            m = Y_hat.shape[1]
            cost = -1 / m * (np.dot(Y, np.log(Y_hat).T) + np.dot(1 - Y, np.log(1 - Y_hat).T))
            return np.squeeze(cost)

        def get_cost_derivative(self,Y_hat,Y):
            return  - (np.divide(Y, Y_hat) - np.divide(1 - Y, 1 - Y_hat))

        def train(self,inputs,labels,epocs):
            for epoc in range(1,epocs+1):
                if epoc%100 is 0:






So far I have tried to implement this as shown above. But there seems to be a mistake because, after training, the network doesn't seem to have learned. Please let me know if you have any inputs.

  • 1
    $\begingroup$ You're confused. These matrices are your weights. In a neural network, the weights are the values of the connections between the neurons. So, you will update these matrices using back-propagation and gradient descent. To update them, you need first to compute the derivative of your loss function, which in this case is the binary cross entropy, with respect to each of these matrices (this will be the gradient of the loss function). $\endgroup$ – nbro Jun 19 '19 at 21:19
  • $\begingroup$ @nbro I have updated the question with my current implementation. Please let me know your inputs. $\endgroup$ – deep_learner Jun 23 '19 at 21:44

The back propagation must be done in two steps.

  • Transfer derivative

    Given the output value, you must calculate its slope. This is simple as you are using the sigmoid activation function. So the derivative can be calculated as follows

# Calculating slope
def transfer_derivative(output):
    return output * (1.0 - output)
  • Back propagate the error

    Calculate the errors for the various layers of the network and update the weights and bias.

# Error calculation

# For output layer
error = (expected - output) * transfer_derivative(output)

# Hidden layers
error = (weight_k * error_j) * transfer_derivative(output)

Where error_j is the error signal from the jth neuron in the output layer, weight_k is the weight that connects the kth neuron to the current neuron and output is the output for the current neuron.

And finally updating weights

weight = weight + learning_rate * error * input

Here is a detailed explanation of implementing neural networks from scratch using Python along with sample code. Give it a read.

|improve this answer|||||
  • $\begingroup$ Hi. I appreciate your help and I encourage you to keep trying to help. However, you're answering to a question where the OP is confused, and I don't think you're addressing his confusion. See my comment under his/her question. $\endgroup$ – nbro Jun 19 '19 at 21:23
  • $\begingroup$ @skillsmuggler I have updated question with the current implementation. Let me know your inputs $\endgroup$ – deep_learner Jun 23 '19 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.